Moral Relativism, or, How I learned to stop worrying and love the bomb

As I write this, I see many people on the twitters and facebooks posting a quote supposedly from Martin Luther King Jr, which posits:

“I mourn the loss of thousands of precious lives, but I will not rejoice in the death of one, not even an enemy. Returning hate for hate multiplies hate, adding deeper darkness to a night already devoid of stars. Darkness cannot drive out darkness: only light can do that. Hate cannot drive out hate: only love can do that.”

Not forgeting Gahndi’s

“An eye for an eye makes the whole world blind”.

Or Yoda’s

“Fear is the path to the Dark Side. Fear leads to anger. Anger leads to hate. Hate leads to suffering”

The US spent 14 years and over $1 Trillion finding Bin Laden, and surrrprise surprise, he was shot while resisting capture. “Dead or Alive”, as George W put it. All we can do now is move forward and make change you can believe in, erm, amends, and try to not let it happen again. Yeah, right.

Yoda was spot on, for the record.

It probably goes without saying for most of you reading this, that you are aware that morality is relative, based on where you’re standing. Some people though, don’t agree/quite remember that Right and Wrong are moral choices. For the most part, it is difficult to judge people based on morality other than your own. Sure, as society we have moral standards, but many are not rigidly black and white, there’s a lot of gray.

Given the audience I have for this post, I’ll put this out here – chances are, you’re not American. Some of you are, sure, but you’re more likely to be either Australian, Canadian or “other” (Hi Rosie!). The fact is, they are a different culture. Sure, there are plenty of similarities, similar language, but we do have individual identities. Thankfully.

As a trope, many people think of blood magic as inherently evil, regardless of the intent.

Many people think that about celebrating a death too, regardless of the victim. It’s just something you _don’t do ever_.

And you’re allowed to think that, regardless of why. Such is your own individual moral choice. But this is one of those areas for me where it’s a grey area of morality. Celebrating a death isn’t going to make someone more dead. How many layers of abstraction do you need to place between a death and the results of someone’s passing before it becomes ok to say that the change is good? Does that abstraction really do any good? Aren’t they still morally the same?

What is the difference between saying “I’m glad SEAL team six launched a 40 person assault on a compound in Abbotobad Pakistan and shot Osama Bin Laden in the eye” and saying “I’m glad that Al-Qaeda no longer has a definitive figurehead”? I see them as equivalent, if somewhat differently phrased ways of portraying the same thing.

Let me Godwin this out for you: how is “I’m glad the Nazi’s were defeated” different from “I’m glad allied troops took back Europe from Nazi forces by killing anyone who didn’t surrender?”.

If you launch a war of aggression, and murder thousands of innocent civilians, you and I clearly have some different opinions on how to influence people and what is right and proper behaviour, and I’m happy for you to be dealt with under your own moral code.

I’m sad that they were unable to capture Osama Bin Laden and bring him to trial, however his moral views would have had no problem with the manner of his death, and if he didn’t mind, I don’t believe we should either. There are more important things to address now. Reducing everything to soundbites and metaphor reduces the ability to rationally discuss, but.. “live by the sword, die by the sword”. He styled himself as a warrior, who brought death to innocents, on the assumption that he would one day die, preferably in battle.

I saw comments calling for his body to smeared with pigs blog and strung up outside the White House. Some people over-react, and I’m happy with calling them out for it to. The US did the right thing in giving him a mostly religiously appropriate burial. In much the same way that the location of Hitler’s death is now a parking lot, you would not want his burial location to be used as a shrine. It showed their respect for his religion, even if he corrupted that religion for his own ideals.

The actual operation raises a number of questions – particularly about Pakistan’s sovereign rights to not have US armed forces kill people inside its borders, but the dynamic of terrorism and how to fight it is not one that respects diplomatic norms. I’m sure the US will say sorry and promise that if they ever have to catch-or-kill Osama Bin Laden again, they’ll ask first. But they’re not going to say anything about Mullah Omar. It’s all very well to use the Niemoller quote of “at first they came for the X [..] then when they came for me, there was no-one left”, but it’s not a slope that is that slippery. For anyone else, they’ll just get your government to extradite you.

The information that lead to this assault apparently came out of a US detainee. I don’t know where, and I don’t know how. I’m hoping it wasn’t torture, because there are fates worse than death, and that is one of them. We are the good guys, we shouldn’t do that. I am hoping that the reason Guantanamo Bay was kept open this long was because they got it from someone in there, and they did not want that person talking before they could act on it. Wikileaks very nearly gave the game away, last week releasing a report mentioning a courier for Osama Bin Laden based out of Abbottabad.

I’ll put it out there: I’m against the death penalty, I’m not in favour of wars of aggression, and I’m glad he has been “neutralised”, however it occurred. In this case, I think the ends justify the means, as they sometimes do, and that’s a moral call on my part. I know I couldn’t be the guy going in with the SMG killing people, but that’s why I’m not in the military. This is also why we have a military, to do distasteful things on our behalf.

Were I in Obama’s place and had the head of the CIA come up to me and say “We’re 99.99% sure we’ve found Bin Laden. Can I send a 40 man team into Pakistan to capture or kill him?”, I’m confident I would say yes. But that doesn’t make it any easier.

Conventional wisdom/morality is easy to understand because we’ve seen the paths that decisions lead in, the normal ones and the outliers. We know the risks. The problem is that there are situations where normal morality doesn’t apply easily, because the questions are so uncommon that there is not an adequate amount outcomes to evaluate. The classic questions here are the ticking timebomb/torture scenario and survival lottery scenarios. By not taking the morally reprehensible option, you are making things far worse than the consequences of taking the option.

I’m glad that the news of his defeat has helped give closure to many in the US and around the world who were affected by his acts. If 300 of your co-workers were killed on someone’s orders, wouldn’t you be at least a little bit glad to see him gone? How about 3000 people from your town? Your city? At what point does it become not ok?

Bin Laden was more than a person to many on both sides of the equation. The inability to catch or kill him gnawed at the western world, but particularly the American consciousness. He was the boogeyman strawman in so many positions taken by people, both good and bad, to justify their actions, both good and bad. I have great anger towards him for the measures our governments have taken in his name that erode our freedoms and civil liberties. I’m not thrilled with them either, but we have elections, a way to theoretically make them change. I am glad he now cannot be used as an argument to influence, as it will be a harder sell to continue them in the future.

The biggest surprise to me was as Obama announced the details of his death. I was expecting something along the lines of “grave/bones found/dna testing complete”. I thought he was long dead. The fact he was killed in the last 72 hours is something I can’t get over.

What I’m really looking forward to is what happens next. We’ve killed the bogeyman. Job well done, now let’s get rid of all the blankets we put up to keep him out.

Recovering from VMDKs on NetApp NFS Datastores

Ok, so the last post went over the scenario of recovering entire VMs, what if you just want one file? As I mentioned, we used to recover the whole VM to another place, copy the file out, then delete our copy. But that was far from elegant, and again, a pain if the file they wanted was in a snapvaulted location.

How much do you trust the filesystem to be consistant? Well, we take a “crash-consistant” snapshot every morning, where the NetApp system effectively spools off a version of the underlaying VMDK file, without telling the Virtual machine using it. Our recovery rate, over the last two years, and 1050 VMs, has been 100%. It’s not a solution for everyone and everything – for the VMs running high transaction load DBs, like Oracle (yup, we went there!) and Exchange, we use NFS or iSCSI, and use NetApp’s Snapmanager products to quiesce the Applications and take snapshots of their storage in the instant they are flushed.

So our crash consistant snapshots, how do we get files back out of them? Remember the secured recovery console VM in the previous post? Remember the inception reference in the previous post? Add a few more layers into that.

The basic premise is that we mount the NTFS filesystems in the VM, using NTFS-3G, and use e2tools to copy files out of ext3 partitions.

But to get to those points, you have a few problems. The first is to turn your read-only VMDK (NetApp snapshots are read-only) into a device. losetup -r loop0 /path/to/VMDK will do that. Then, find partitions inside this device: kpartx -a -v /dev/loop0. At this point, you can just mount the NTFS partitions from the Windows VMs, but the Linux systems have a few more tricks up their sleeves..

We use LVM, for flexible volume management. It’s burnt into our template. Which means all of our VMs have the same VG and LV names. The first thing we did to prepare this recovery VM was to rename it’s Volume Groups to avoid conflicts. Simple vgchange, edit /etc/fstab and mkinitrd – in that order. If you do mkinitrd before the /etc/fstab edit, the initrd will load root from a non-existant location.

Having prepared our recovery VM in advance, we scan for volume groups inside the /dev/loop0 partitions using vgscan, then bring them online with vgchange -ay VGname

At this point, you’d think we could just mount the LVs, wouldn’t you?

Quick primer on the ext3 filesystem – it’s ext2, with a journal to enable easy recovery after crashes. In these crash consistant VMDK snapshots, there’s an unflushed journal, and the filesystem is flagged as inuse and having one. Linux’s ext3 implementation will attempt to replay the journal of an ext3 filesystem if present, when mounted. Even if you tell it not to load the journal (noload), it will still attempt to make your readonly filesystem read-write to mark the filesystem as clean. And if you try to mount it as ext2, it will also complain, since there’s a journal there. ext3 journals can be removed, but guess what? It’s a read-write operation. All of these things are perfectly reasonable, and there for very very good reasons. Just, not what I’m after, since this is a 100% read-only situation, and I can’t make it readwrite, even if I wanted to.

So we looked at a couple of options, union filesystems (rejected; wanted to copy the whole VMDK if we made a change), guestfish (actually works ok, but is very resource heavy – it essentially boots the VM inside it) and eventually were pointed at e2tools – it’s in early beta, and it hasn’t been updated in 7 years – but it seems perfectly functional.

At this point, we’ve copied our files out, with just cp or e2cp, so how do we get them to the VM? We’re still working on that, but current plan is to use mkisofs to turn them into an .iso, and mount that to the VM for the end-admin to copy them out of.

Then, once all the copies are done, you need to tear down the LVM with vgchange -an, delete the partitions from the kernel with kpartx -d, then remove the loop device with losetup -d and you’re done! We will be automating a lot of this with some shell scripts (think – startrecover, stoprecover to take care of the loop/LVM setup), but even now it’s a lot quicker than what we had.

Pretty neat huh?

Recovering VMDKs on NetApp NFS Datastores

In my day job, I look after the day to day server operations of a university that makes extensive use of vmware and netapp storage. When I started there, and saw they were using NFS for their datastores, I reversed judgement on if they were crazy-smart or just crazy. Thankfully it was the former – crazy-smart.

Using NetApp NFS for VMDK storage allows us to do all sort of cool stuff, especially with regards to backups/recovery/migration. But it had been tedious, especially if someone wanted a single file restored from their VM.. we had to copy the entire VMDK out of the snapshot directory, mount it on another VM somewhere, find the file, and get it back to the customer somehow. And if it was on our secondary filer, we had to do a flexclone, and mount that onto one of the 96 ESX hosts we had, copy the file out.. etc

Wheels spin sometimes, and an idea comes to you. Remember Inception? and all the layers? Going deeper etc? It’s like that.

/home/user/file.txt -> ext3 -> LVM LV -> LVM VG -> LVM PV -> /dev/sda1 -> ESX -> VMDK -> NFS Datastore -> NetApp Data OnTap -> WAFL -> Disks ..

Over the last week, my co-workers and I have been building up a system to make this easier and less disruptive to the infrastructure (which is good for everyone, the less changes you have to make to production, the better). This gist is this..

We have a secured VM, with a couple of NICs – one standard access port, one a VLAN trunk carrying our NAS networks, including the one that the VM Blades use to mount their storage.

Inside this VM, we do magic…

So, 96 blades – that’s a fairly large VM infrastructure. We have two separate environments, in 6 clusters, two routing domains, etc, running a total of 1050+ VMs at last count. Each cluster with their own datastores, diverse physical locations, etc. One of the service improvement projects that I got our great team to do was to implement were some datastores, mounted onto all the clusters, routed where needed. Performance didn’t have to be great, just good enough, and on 10Gb NFS, yeah, it’s pretty good. We have an ISOs datastore, a Templates datastore and a Transfer datastore. The Transfer one was new – the others we’d had for a while.

On our secured VM, we have the Transfer datastore mounted read-write using NFS, as well as the snapvault repository versions of our datastores (mounted read only for safety, but the files are read-only anyway). This now means that if we have to do a full VM recover, we have a simple process –

  • Shut down the VM
  • Edit the settings to remove the hard drives you want to recover (I know, it sound wrong to me too, but trust me..)
  • Storage vMotion the VM onto the Transfer datastore (which, since it doesn’t have any disks, is quick)
  • Locate the version of the VMDK you want in the .snapshot directory of the snapvault location (We have a simple shell script to list all versions)
  • Copy the VMDK files (remember the -flat.vmdk) from the snapvault location into the appropriate directory on the Transfer datastore, using cp &, then running watch ls -l on the destination, if you want a progress indicator
  • Re-add the storage from the vmware settings, finding it in the place you just copied it
  • Power On VM, check it works, then hand back control to customer, and start a storage vMotion to relocate storage back into the correct primary datastore

All done! No messing around on the NetApp making flexclones and mounting them, cleaning them up etc. Depending on your level of risk tolerance, you could copy the VMDK back to the primary location also mounted via NFS, but we consider the small delay of the storage vMotion to be a price worth paying for peace of mind.

Your site gets compromised, what do you do?

.. make people unable to use authentication methods that don’t involve giving you a password, that’s what!

Following on from the Gawker account hack, I have gone and changed a bunch of accounts, even though I may not have actually used a password I generated for Gawker, but it seemed prudent.

Lifehacker have a page up here which details the response..

Including this bit:

2) What if I logged in using Facebook Connect? Was my password compromised?
No. We never stored passwords of users who logged in using Facebook Connect. We have, however, disabled Facebook Connect logins temporarily.


So what you’re saying is, not only are you incompetent, and let people steal your user/password database, you’ve now stopped the only way of stopping it from happening again??

Nothing pisses me off more than websites that require you to register or login to look at attachments on forums, for example. Facebook Connect (or ideally OpenID) are an awesome solution to the problem of having to create/maintain/worry about accounts on every site on the internet. I mean sure, there are idiots in marketing who love the idea of “rich user engagement” from tying them to your site with an account, but I think they severely overestimate their own importance.

.. but don’t get me started on janrain/rpx’s recent change that suggests you put your paypal username/password into HTML hosted on an insecure site so you can join the social engagement “story”. That’s just stupid.

Fixing GPT partition tables for OSX

With our upcoming visit to Australia, we’re doing backups before we go away. But alas! Elizabeth’s USB drive didn’t work. It became unmounted, and when she plugged it back in, no volumes were found!

.. by OSX

Never wanting to throw away the contents of a drive, I started digging. On a Linux box, I used parted to look at the disk and find that it did indeed know about all the partitions that should be on there, but for whatever reason, they weren’t being enumerated.

Satisfied the data was still there, I went back to my Mac and started poking around. I could see that /dev/disk1 existed, and had no partitions, just as OSX would have be believe. Using the gpt command line utility, I got the following:

# gpt -r show -l /dev/disk1
start size index contents
0 1
1 1 Pri GPT header
2 32 Pri GPT table
34 6
40 409600 1 GPT part - "EFI System Partition"
409640 1464471472 2 GPT part - "Time Machine Backups"
1464881112 262151
1465143263 32 Sec GPT table
1465143295 1 Sec GPT header

Twirling my evil moustache, I thought if I could relabel one of those partitions, it would make it rewrite both partitions, and she should be apples.

# gpt label -i 2 -l "Time Machine Backups" /dev/disk1
/dev/disk1s2 labeled

But no. I then wondered if /usr/sbin/diskarbitrationd was saying anything helpful about the situation, and ran it in debug mode (edited /System/Library/LaunchDaemons/ as root to add the -d flag to startup), and then kill -HUP `cat /var/run/` and then tail -f /var/log/diskarbitrationd.log and I got this:

18:11:14 probed disk, id = /dev/disk1, with cd9660, failure.
18:11:14 probed disk, id = /dev/disk1, with exfat, ongoing.
18:11:14 probed disk, id = /dev/disk1, with exfat, failure.
18:11:14 probed disk, id = /dev/disk1, with msdos, ongoing.
18:11:14 probed disk, id = /dev/disk1, with msdos, failure.
18:11:14 probed disk, id = /dev/disk1, with ntfs, ongoing.
18:11:14 probed disk, id = /dev/disk1, with ntfs, failure.
18:11:14 probed disk, id = /dev/disk1, with ufs, ongoing.
18:11:14 probed disk, id = /dev/disk1, with ufs, failure.
18:11:14 probed disk, id = /dev/disk1, no match.

Good effort though, right? I mean, I’m sure Apple must expect regular users to put diskarbitrationd into debug mode on a regular basis.


Found out from this blog post that gdisk was available for OSX. Downloaded, installed and ran it:

# gdisk /dev/disk1
GPT fdisk (gdisk) version 0.6.13

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: present

Found valid GPT with corrupt MBR; using GPT and will write new
protective MBR on save.

Command (? for help): ?

To avoid prolonging the story any more, wrote the partition table to disk, and hey presto, there’s all the data back.

So what did we learn from this? Neither Apple, nor Linux, will try using a backup GPT if the primary one becomes fubared.

And despite all assurances to the contrary, USB bus-powered 2.5 inch HDD’s only just work with OSX’s meager power provision, and if they get unplugged, they won’t have enough juice to flush caches.

so BC is getting a new Premier

The big news in BC yesterday was that Gordon Campbell stepped down as Premier. Some were loudly proclaiming victory, or expressing happiness of his departure.

As he put it in his statement: When public debate becomes focused on one person, instead of what is in the best interest of British Columbians, we have lost sight about what is important. When that happens, it’s time for a change.

Cause let’s look at the mess he left BC in:

  • One of the lowest unemployment rates in Canada.
  • third highest average hourly wage in Canada
  • lowest tax rate for low-income (0%) and middle-income families in Canada.
  • up to a 70% tax reduction for low income families
  • opened 80 new schools, increased education funding every year, more seats in universities, highest per-pupil funding in Canada
  • Balanced budgets for 9 years until the biggest recession in half a century.
  • 42% reduction in provincial budgets before service cuts
  • A provincial credit rating that has been upgraded 7 times in a row to AAA (the highest possible)
  • biggest real GDP growth in Canada
  • $195 million in new Arts grants
  • $80 million in new permanent sport grants and funding
  • 20% increase in the amount paid per person by income assistance
  • Low-income support program spending up by more than 4x
  • Reduced carbon and greenhouse gas emissions – the most aggressive targets set in Canada, with legal enforcement in place
  • (ganked from voice_of_experience on reddit)

    Oh wait, that’s the good stuff.

    And yet, there’s a downside.. apparently some people don’t like the HST (which, when you look at what else the province gives, is actually a reasonable measure..) or they didn’t like the Olympics (what are you going to do about that now? it worked out fine. Sure it cost a lot of money..) or that the Canada Line doesn’t have enough capacity (it grew faster than expected, that’s success isn’t it?), or that he once got arrested for drink driving (let me tell you about Ralph…)

    I’m confident of history’s view of this period in politics. Also, has anyone seen Idiocracy? No? Never mind, seems like it’s playing out in politics right now, what with this and the Tea Party..

    Australian food

    We went to a place called Moose’s Downunder for lunch on Sunday, who bill themselves as providing a little bit of home and a unique Australian experience in Vancouver.

    Well it’s certainly as described on box. It seems to be staffed entirely by Australians, many of whom are from Perth like the owner. I had an Aussie Burger, with Beetroot + Fried Egg + Pineapple. It did indeed remind me of home. Also the chairs were EXACTLY the same as the ones that KK’s/The Last Drop in Crawley used to have before it turned upmarket. Down to the varnish on the arms turning gooey and coming off.

    On the downsides, just like home they charge for drink refills and extra sauces. So just like home, you don’t have to tip, right? :P I kid, I kid. I did tip, as is the local custom.