I could write a lot about Steve Job’s death, but a lot of it has been said by others, and said well.

I’ll take the anonymous proverb, sometimes mis-attributed to Dr Seuss – “Don’t cry because it’s over, smile because it happened.”

And what happened was amazing – from the “ignition” (as their press releases say) of the personal computer market, their decline, NeXT’s rise, Apple’s return and to the sheer scale of their success.

But I’m sad because it wasn’t finished. He won’t see the mothership built, or even his new house, or the growth and development of his youngest children, or the evolution of Apple or society.

I’m sorry he didn’t get a dotage – a relaxing retirement. I’m sorry he didn’t get to do more philanthropic work. I’m sorry he wasn’t able to make Apple a good corporate citizen – the ~$80 billion Apple has in the bank is money that has come from consumers and hasn’t “trickled down” back into society – and that’s a big problem.

But, for all these things he wasn’t able to achieve in one lifetime, he still did some truly amazing things.

.. and all I got was this stupid t-shirt.

Actually, I didn’t get a t-shirt.

Despite long layovers getting there, the TSA and lots of humidity, we had a great time.

Day 0:

Left Vancouver in the afternoon, flew to LAX. Yay pre-clearance of US customs and immigration in Canada.

LAX is.. almost indescribable. You have to make an effort to make an airport this bad. We arrived and had to change terminals. You’d think that’s reasonably common right? The sort of thing you’d put signs up saying how to do. Guess again!

We had a few hours in LAX, then flew to Miami (MIA). The LAX/MIA flight was 100% full, every overhead bin, every seat. Also, “overnight” and a 4 hour time change forwards, so we arrived in MIA at 1AM Vancouver time, 5AM Miami time. Sleep? Ha! But to make it easy on us, they played Rio on the overhead LCDs – so good luck trying to sleep with bright flashing colours in your eyes.

Liz and I got a few hours sleep on the floor at MIA before getting up to find some breakfast and catch our plane to San Jose (SJO). Which is a 3 hour change backwards.

Day 1:

We met up with Elizabeth’s sister, Caitlin, in the throngs of people at San Jose Costa Rica airport, and got a pre-arranged shuttle to the apartment hotel I booked for us.

Checked in, showered, then slept a bit (yay disrupted sleep!). E+C went out and got salsa, refried beans, corn chips and tortillas from a nearby shop, and we had some snacks. Did lots of chatting and planning, planned a trip to a butterfly farm for the next day. Had Quiznos for dinner – ordered in spanish.

Day 2:

Went to butterfly farm. It was in an urban-fringe type area, and I was not disappointed in my stereotypes of central America. Most of the buildings had tin roofs, and wandering chickens and dogs. Also, the roads had giant pot holes.

Day 3:

Took a day trip through the countryside. Included a river boat trip spotting animals, authentic Costa Rican lunch and a ski-lift/sky-tram through the forrest canopy. Nice and relaxing.

Day 4:

Went to downtown San Jose. Plans was to check out the gold museum and banking museum. Unfortunately they were both mostly closed for renovation. Downtown San Jose is.. not very nice. Sort of, get-out-of-town-before-sunset type not nice. But we survived ok, and caught public buses in and out. It was an experience.

Day 5:

Tour to Irazú volcano. Saw volcano – no lava though. Got irascible at Irazú due to misunderstanding of meeting places and subsequently having to expend a lot of energy at high altitudes. But it all worked out in the end. Went to Lankester Gardens, took many photos of flowers.

Back at the hotel, tried doing laundry. Dryer didn’t work, but the staff pre-emptively fixed that by drying it in their industrial dryer and returning it to us mostly folded. Yay!

Day 6:

Tour to Monteverde. Driver was an hour late, then we got stuck for almost an hour on the pan-american highway due to protestors. Admired Police for their restraint and for protecting the protestors – I would guarantee doing the same thing in Australia would get you pepper-sprayed and hauled off in flex-cuffs quicker than you could say “We want..” Drive to Monteverde was up 15kms of an amazingly bumpy dirt road in a Toyota Hiace. It took almost an hour up, and 30 minutes down. Went on a walking tour through forrest and over suspension bridges. Saw a Queztel, but didn’t have my 400mm lens on, so only got a crappy photo. But it was fun :)

Day 7:

Packed and off to the airport. SJO-MIA-LAX. No US Customs pre-clearance, so arrived in Miami and had to do a massive trek through the terminal to get to Customs, then waiting in line for an hour. Yaaaay. Then re-check bags, re-clear security, get to gate with a few minutes to spare, wolf down dinner.. then the plane to LAX is delayed by an hour due to lightning. Doh! Uneventful flight to LAX, picked up at airport by E+C’s cousin Nadine, dropped off at hotel near her place, shower, and sleep by 2AM.

Day 8:

Photos from LA. Breakfast at hotel, shared table and chatted with couple from Teax. Walk to beach, meet Caitlin (staying at cousin’s place near beach). Wander around, watch the beginning of Smackfest, a beach volleyball tournament. Back to Nadine’s place, then back to beach where C+N swim, Elizabeth and I watch and soak up the rays, have some brunch/second breakfast, meet up with Nadine’s husband Anthony. Back to their place, then brave LA highways, go to lookout over Hollywood sign. Brave more freeways, tourists, drive through Sunset strip, Rodeo drive, Beverley Hills, go to La Brea Tarpits. Dinner at swanky little restaurant, back to N+A’s place, chat till 11PM, walk back to hotel.

Day 9:

Mini-breakfast at hotel, including made on the spot waffle (om nom nom). Checked out and picked up from hotel, brunch again with C+N+A. Went to BestBuy, picked over the carcass of a Borders store. Had ice cream. Hung out and chatted with them some more, then off to the airport, sad goodbyes with Caitlin and flight back to YVR. In bed by 11:30PM, and then up for work the next morning!

Summary:

Wonderful to see Caitlin again. Costa Rica was as imagined and really nice. Packed efficiently.

Over the last few months, I’ve really become a big fan of Perth boy Tim Minchin. Of all the ways I could have really looked into his music, I was reminded of my desire to look into it by a co-worker of mine here in Vancouver.

Being relevant to my interests, I found out he was playing in Seattle, and was initially in a quandary, since it was at the same time as another show that Elizabeth wanted to see on the other side of Seattle. Thankfully he put on a second show, so we both went to the one Elizabeth wanted to see on Friday, then I went to Tim’s show last night.

So I spent my Saturday night in the Neptune Theater of Seattle, with about 600 other people, watching the son of a high school friend of my mother’s as he played:

• Rock and roll nerd
• Cont / context (Unreleased)
• If I didn’t have you
• Thank you god (Unreleased)
• Lullaby
• Prejudice
• The Pope song
• Confessions
• Storm
• Darkside
• Take me home country roads (short)
• I will survive (short)
• White wine in the sun
• Hallelujah

He’s an incredibly talented musician and pianist, and has amazing comic timing too. In the stilted timings of a guest of Adventure World in the early 2000s: “It was really good.”

Part the first – des vacances

– We’re going to Seattle on the weekend, and seeing Panic at the disco and fun on friday night, and Tim Minchin on saturday night. Awes. May also see Ebbie, for the first time since 1999.

 

– In mid July, we’re going to Costa Rica, to meet up with Elizabeth’s sister, since it’s a lot closer than Canberra, and to see some of central America. It’s our first time going to a non-first world nation, and today we’re getting poked with various engineered compounds to keep us well.

 

– In August we’re planning a weekend away in Oregon

 

– In October, we’re planning on a visit to NY (don’t think any of you live there?) and Washington (sup Stephanie  and Claire!)

 

– In December, Elizabeth’s parents and brother and sister are coming to visit for Christmas. Plans are for the Island somewhere. We should book that soon.

 

Part the second – nous voiture

Following an unexpected encounter with a bike trailer, ‘er majesty’s Crown Corporation, The Insurance Company of British Columbia saw it fit to pay for a good whack of repairs to our car, and in for a penny, in for a pound, we also paid to get some other stuff fixed, original paint damage from before we bought it, and a few dents from not being quite used to its size/shape in relation to our parking garage.

 

It took a week longer than quoted, the car smells of paint, and the paint job is definitely not concours standard, but it is a whole lot better than it was.

 

For those interested in the source of the original covered damage, we were driving along one day, and a guy on a bike on the footpath was riding in the same direction. As we passed him, the rope fixation of his trailer came loose, and it fell into the road and onto our car, scratching from passenger mirror to rear bumper. Yeah. We stopped and talked to him, and the trailer had contained all the cooking and camping gear he used living on the streets. Feeling sorry for him and glad he was well, we went on our way.

 

When we bought the car, the hood (bonnet) and front bumper were covered in paint chips, and the day after we got it, I backed it into a pillar in our parking spot, injuring the rear bumper, quarter panel and cracking the tail light assembly. Then, while heading off to Whistler with Raif in August 2010, I misjudged another pillar in our garage and scraped the rear door and quarter panel. Awesome, huh? We had a quote to fix all of this stuff for $1400. With the covered damage from the bike trailer, it was only $1100 to fix it, but since the other “driver” could not be contacted, it was a $300 excess, or .. $1400 in total. But at least I get $15 back from my 1% cash rewards credit card.

 

D’oh!

 

Part the third – en été

 

It’s summer! Trees are blooming, birds are singing, it’s not longer 10C and rainy. What a time to be alive!

 

We went out for dinner last night at our local Cactus Club Cafe, and sat on the “patio”, as Canadians so politely term it. As it was also the equinox, despite it being almost 7PM, it was still broad daylight. As it was until 9:30PM. Thanks Canada. Thanada.

 

Part the forth – le immigration

 

There has been a postal strike in Canada for the last week or two, so no mail. Apparently it’s being sorted out (through legislating them back to work.. possibly the only option, given some of the demands were less money and more totally reshaping Canada Post’s business model to keep existing staffing levels in face of declining mail volume)

 

This is related since it will soon be 20 weeks since we submitted our Permanent Residency applications, which is the timeframe for initial notification of receipt of application. We sent it registered mail, and you can see on the website that it was accepted by Immigration Canada, but this is the one where they tell us we completed the craft project successfully. Hopefully.

 

This weekend also marks 2 years since we arrived here in Canada. It’s been a lot of fun, so we’re sticking around for now. I still miss the old country, and eagerly await Nic Mayer’s return, hopefully with Minties and musk sticks. On Saturday we found the last White Knight we brought back from Perth in Jan (it had been in the freezer), and we nommed it good and proper. We also cut up a fresh pineapple last night. It’s all gone now.

 

But it’s not just the food I miss, it’s the people too. It was sad not to be able to go to Helen’s wedding, and seeing Alexa in a video of it made me a bit nostalgic, and it’s been sad not to be around for Matt and Susie’s first child being born.

 

More of you should add me on Skype – I’m theducks.org on there :P

 

Yesterday was National Aboriginal Day here in Canada. This is related, since 12 years ago I was also in Vancouver, and remember going to an exhibition in Robson Square. If future me had come up to me at the time and told me that in 12 years time I was going to be living in Vancouver, I would have been somewhat surprised, and my initial guess at possible reasoning would have been way off.

 

As I posted on twitter some time ago: “Short version: Adult life turns out fine. You’ll never see where I am now from where you are now, but you like surprises, right? #tweetyour16yearoldself”

 

Part the fifth – le travail

 

Still the Manager of the Systems team, all 16 of them (14 of whom are older than me). I suppose it gets to a point where that doesn’t matter, and I suspect I’m approaching that, but it’s still a change from Arts, where of the 21 in my 3 teams, only 7 were older than me. We’re interviewing for a new Manager this week and next, which should be good. I’m on the interview/selection panel for it, and I’m not motivated to apply for it, partially because it’s only $50/week more, but also because it would mean a definitive move away from tech, and I don’t think that’s for the good of the organisation or really what I want right now.

 

I read through about 65 resumes for 3 jobs yesterday. Too much braining. But being on the hiring side of things does give you a good insight into how to write resumes. Looks like some good candidates there. I seem to have ended up on a lot of hiring committees this year. It’s nice to be trusted, but I sometimes wonder if I volunteer because I don’t trust other people to make these sort of decisions. Then I remember our helpdesk guy who insisted he could only give me a USB switch, not a USB hub, but that would be ok, since it was faster. So it’s clear some people can’t be /o\

 

It has been mandated that we start tracking how staff time is spent, in broad categories like maintenance, projects, admin and leave. Where have I heard that before? Hmm! Today I got to break the news to my team that we need to do it, and defend something that I know in their position I’d probably be initially peeved about. But I think it’s a good thing and I think it went down well.

 

Ahh yes, UCS, circa 2005. I wrote an application back then to do it (remember timekeeper guys? :). I found a copy of it all, except the SQL schema, in a uuencoded attachment. So I’ve dusted it off, and mostly rewritten it (No longer requires Tcl/tk! now has client authentication! No longer requires pop3 auth to cyllene to login!) I hope that showing off this application and offering it to them will help with acceptance, as well as explaining the motivations behind the reporting requirement – transformation goals of moving from being reactive to proactive, avoiding overloading us with project work.

 

It was also fun to do some PHP again, after 3 years since I did much, and 6 years since I did a medium sized app like this. Elizabeth helped, and showed me in detail something she’d done for her team. jquery is pretty awesome, even though I didn’t use it in mine.

 

In other news, we ipv6 enabled the university’s main website, and I took the opportunity to setup ipv6 on my linode host, so theducks.org is also ipv6 enabled. I just need to make like an animated TRON duck or something to display to ipv6 viewers.

 

So that about wraps up this mega update..

#604insomnia, brought to you by Coca-Cola with dinner. A bad idea since 1886.

As I write this, I see many people on the twitters and facebooks posting a quote supposedly from Martin Luther King Jr, which posits:

“I mourn the loss of thousands of precious lives, but I will not rejoice in the death of one, not even an enemy. Returning hate for hate multiplies hate, adding deeper darkness to a night already devoid of stars. Darkness cannot drive out darkness: only light can do that. Hate cannot drive out hate: only love can do that.”

Not forgeting Gahndi’s

“An eye for an eye makes the whole world blind”.

Or Yoda’s

“Fear is the path to the Dark Side. Fear leads to anger. Anger leads to hate. Hate leads to suffering”

The US spent 14 years and over $1 Trillion finding Bin Laden, and surrrprise surprise, he was shot while resisting capture. “Dead or Alive”, as George W put it. All we can do now is move forward and make change you can believe in, erm, amends, and try to not let it happen again. Yeah, right.

Yoda was spot on, for the record.

It probably goes without saying for most of you reading this, that you are aware that morality is relative, based on where you’re standing. Some people though, don’t agree/quite remember that Right and Wrong are moral choices. For the most part, it is difficult to judge people based on morality other than your own. Sure, as society we have moral standards, but many are not rigidly black and white, there’s a lot of gray.

Given the audience I have for this post, I’ll put this out here – chances are, you’re not American. Some of you are, sure, but you’re more likely to be either Australian, Canadian or “other” (Hi Rosie!). The fact is, they are a different culture. Sure, there are plenty of similarities, similar language, but we do have individual identities. Thankfully.

As a trope, many people think of blood magic as inherently evil, regardless of the intent.

Many people think that about celebrating a death too, regardless of the victim. It’s just something you _don’t do ever_.

And you’re allowed to think that, regardless of why. Such is your own individual moral choice. But this is one of those areas for me where it’s a grey area of morality. Celebrating a death isn’t going to make someone more dead. How many layers of abstraction do you need to place between a death and the results of someone’s passing before it becomes ok to say that the change is good? Does that abstraction really do any good? Aren’t they still morally the same?

What is the difference between saying “I’m glad SEAL team six launched a 40 person assault on a compound in Abbotobad Pakistan and shot Osama Bin Laden in the eye” and saying “I’m glad that Al-Qaeda no longer has a definitive figurehead”? I see them as equivalent, if somewhat differently phrased ways of portraying the same thing.

Let me Godwin this out for you: how is “I’m glad the Nazi’s were defeated” different from “I’m glad allied troops took back Europe from Nazi forces by killing anyone who didn’t surrender?”.

If you launch a war of aggression, and murder thousands of innocent civilians, you and I clearly have some different opinions on how to influence people and what is right and proper behaviour, and I’m happy for you to be dealt with under your own moral code.

I’m sad that they were unable to capture Osama Bin Laden and bring him to trial, however his moral views would have had no problem with the manner of his death, and if he didn’t mind, I don’t believe we should either. There are more important things to address now. Reducing everything to soundbites and metaphor reduces the ability to rationally discuss, but.. “live by the sword, die by the sword”. He styled himself as a warrior, who brought death to innocents, on the assumption that he would one day die, preferably in battle.

I saw comments calling for his body to smeared with pigs blog and strung up outside the White House. Some people over-react, and I’m happy with calling them out for it to. The US did the right thing in giving him a mostly religiously appropriate burial. In much the same way that the location of Hitler’s death is now a parking lot, you would not want his burial location to be used as a shrine. It showed their respect for his religion, even if he corrupted that religion for his own ideals.

The actual operation raises a number of questions – particularly about Pakistan’s sovereign rights to not have US armed forces kill people inside its borders, but the dynamic of terrorism and how to fight it is not one that respects diplomatic norms. I’m sure the US will say sorry and promise that if they ever have to catch-or-kill Osama Bin Laden again, they’ll ask first. But they’re not going to say anything about Mullah Omar. It’s all very well to use the Niemoller quote of “at first they came for the X [..] then when they came for me, there was no-one left”, but it’s not a slope that is that slippery. For anyone else, they’ll just get your government to extradite you.

The information that lead to this assault apparently came out of a US detainee. I don’t know where, and I don’t know how. I’m hoping it wasn’t torture, because there are fates worse than death, and that is one of them. We are the good guys, we shouldn’t do that. I am hoping that the reason Guantanamo Bay was kept open this long was because they got it from someone in there, and they did not want that person talking before they could act on it. Wikileaks very nearly gave the game away, last week releasing a report mentioning a courier for Osama Bin Laden based out of Abbottabad.

I’ll put it out there: I’m against the death penalty, I’m not in favour of wars of aggression, and I’m glad he has been “neutralised”, however it occurred. In this case, I think the ends justify the means, as they sometimes do, and that’s a moral call on my part. I know I couldn’t be the guy going in with the SMG killing people, but that’s why I’m not in the military. This is also why we have a military, to do distasteful things on our behalf.

Were I in Obama’s place and had the head of the CIA come up to me and say “We’re 99.99% sure we’ve found Bin Laden. Can I send a 40 man team into Pakistan to capture or kill him?”, I’m confident I would say yes. But that doesn’t make it any easier.

Conventional wisdom/morality is easy to understand because we’ve seen the paths that decisions lead in, the normal ones and the outliers. We know the risks. The problem is that there are situations where normal morality doesn’t apply easily, because the questions are so uncommon that there is not an adequate amount outcomes to evaluate. The classic questions here are the ticking timebomb/torture scenario and survival lottery scenarios. By not taking the morally reprehensible option, you are making things far worse than the consequences of taking the option.

I’m glad that the news of his defeat has helped give closure to many in the US and around the world who were affected by his acts. If 300 of your co-workers were killed on someone’s orders, wouldn’t you be at least a little bit glad to see him gone? How about 3000 people from your town? Your city? At what point does it become not ok?

Bin Laden was more than a person to many on both sides of the equation. The inability to catch or kill him gnawed at the western world, but particularly the American consciousness. He was the boogeyman strawman in so many positions taken by people, both good and bad, to justify their actions, both good and bad. I have great anger towards him for the measures our governments have taken in his name that erode our freedoms and civil liberties. I’m not thrilled with them either, but we have elections, a way to theoretically make them change. I am glad he now cannot be used as an argument to influence, as it will be a harder sell to continue them in the future.

The biggest surprise to me was as Obama announced the details of his death. I was expecting something along the lines of “grave/bones found/dna testing complete”. I thought he was long dead. The fact he was killed in the last 72 hours is something I can’t get over.

What I’m really looking forward to is what happens next. We’ve killed the bogeyman. Job well done, now let’s get rid of all the blankets we put up to keep him out.

Ok, so the last post went over the scenario of recovering entire VMs, what if you just want one file? As I mentioned, we used to recover the whole VM to another place, copy the file out, then delete our copy. But that was far from elegant, and again, a pain if the file they wanted was in a snapvaulted location.

How much do you trust the filesystem to be consistant? Well, we take a “crash-consistant” snapshot every morning, where the NetApp system effectively spools off a version of the underlaying VMDK file, without telling the Virtual machine using it. Our recovery rate, over the last two years, and 1050 VMs, has been 100%. It’s not a solution for everyone and everything – for the VMs running high transaction load DBs, like Oracle (yup, we went there!) and Exchange, we use NFS or iSCSI, and use NetApp’s Snapmanager products to quiesce the Applications and take snapshots of their storage in the instant they are flushed.

So our crash consistant snapshots, how do we get files back out of them? Remember the secured recovery console VM in the previous post? Remember the inception reference in the previous post? Add a few more layers into that.

The basic premise is that we mount the NTFS filesystems in the VM, using NTFS-3G, and use e2tools to copy files out of ext3 partitions.

But to get to those points, you have a few problems. The first is to turn your read-only VMDK (NetApp snapshots are read-only) into a device. losetup -r loop0 /path/to/VMDK will do that. Then, find partitions inside this device: kpartx -a -v /dev/loop0. At this point, you can just mount the NTFS partitions from the Windows VMs, but the Linux systems have a few more tricks up their sleeves..

We use LVM, for flexible volume management. It’s burnt into our template. Which means all of our VMs have the same VG and LV names. The first thing we did to prepare this recovery VM was to rename it’s Volume Groups to avoid conflicts. Simple vgchange, edit /etc/fstab and mkinitrd – in that order. If you do mkinitrd before the /etc/fstab edit, the initrd will load root from a non-existant location.

Having prepared our recovery VM in advance, we scan for volume groups inside the /dev/loop0 partitions using vgscan, then bring them online with vgchange -ay VGname

At this point, you’d think we could just mount the LVs, wouldn’t you?

Quick primer on the ext3 filesystem – it’s ext2, with a journal to enable easy recovery after crashes. In these crash consistant VMDK snapshots, there’s an unflushed journal, and the filesystem is flagged as inuse and having one. Linux’s ext3 implementation will attempt to replay the journal of an ext3 filesystem if present, when mounted. Even if you tell it not to load the journal (noload), it will still attempt to make your readonly filesystem read-write to mark the filesystem as clean. And if you try to mount it as ext2, it will also complain, since there’s a journal there. ext3 journals can be removed, but guess what? It’s a read-write operation. All of these things are perfectly reasonable, and there for very very good reasons. Just, not what I’m after, since this is a 100% read-only situation, and I can’t make it readwrite, even if I wanted to.

So we looked at a couple of options, union filesystems (rejected; wanted to copy the whole VMDK if we made a change), guestfish (actually works ok, but is very resource heavy – it essentially boots the VM inside it) and eventually were pointed at e2tools – it’s in early beta, and it hasn’t been updated in 7 years – but it seems perfectly functional.

At this point, we’ve copied our files out, with just cp or e2cp, so how do we get them to the VM? We’re still working on that, but current plan is to use mkisofs to turn them into an .iso, and mount that to the VM for the end-admin to copy them out of.

Then, once all the copies are done, you need to tear down the LVM with vgchange -an, delete the partitions from the kernel with kpartx -d, then remove the loop device with losetup -d and you’re done! We will be automating a lot of this with some shell scripts (think – startrecover, stoprecover to take care of the loop/LVM setup), but even now it’s a lot quicker than what we had.

Pretty neat huh?

In my day job, I look after the day to day server operations of a university that makes extensive use of vmware and netapp storage. When I started there, and saw they were using NFS for their datastores, I reversed judgement on if they were crazy-smart or just crazy. Thankfully it was the former – crazy-smart.

Using NetApp NFS for VMDK storage allows us to do all sort of cool stuff, especially with regards to backups/recovery/migration. But it had been tedious, especially if someone wanted a single file restored from their VM.. we had to copy the entire VMDK out of the snapshot directory, mount it on another VM somewhere, find the file, and get it back to the customer somehow. And if it was on our secondary filer, we had to do a flexclone, and mount that onto one of the 96 ESX hosts we had, copy the file out.. etc

Wheels spin sometimes, and an idea comes to you. Remember Inception? and all the layers? Going deeper etc? It’s like that.

/home/user/file.txt -> ext3 -> LVM LV -> LVM VG -> LVM PV -> /dev/sda1 -> ESX -> VMDK -> NFS Datastore -> NetApp Data OnTap -> WAFL -> Disks ..

Over the last week, my co-workers and I have been building up a system to make this easier and less disruptive to the infrastructure (which is good for everyone, the less changes you have to make to production, the better). This gist is this..

We have a secured VM, with a couple of NICs – one standard access port, one a VLAN trunk carrying our NAS networks, including the one that the VM Blades use to mount their storage.

Inside this VM, we do magic…

So, 96 blades – that’s a fairly large VM infrastructure. We have two separate environments, in 6 clusters, two routing domains, etc, running a total of 1050+ VMs at last count. Each cluster with their own datastores, diverse physical locations, etc. One of the service improvement projects that I got our great team to do was to implement were some datastores, mounted onto all the clusters, routed where needed. Performance didn’t have to be great, just good enough, and on 10Gb NFS, yeah, it’s pretty good. We have an ISOs datastore, a Templates datastore and a Transfer datastore. The Transfer one was new – the others we’d had for a while.

On our secured VM, we have the Transfer datastore mounted read-write using NFS, as well as the snapvault repository versions of our datastores (mounted read only for safety, but the files are read-only anyway). This now means that if we have to do a full VM recover, we have a simple process –

• Shut down the VM
• Edit the settings to remove the hard drives you want to recover (I know, it sound wrong to me too, but trust me..)
• Storage vMotion the VM onto the Transfer datastore (which, since it doesn’t have any disks, is quick)
• Locate the version of the VMDK you want in the .snapshot directory of the snapvault location (We have a simple shell script to list all versions)
• Copy the VMDK files (remember the -flat.vmdk) from the snapvault location into the appropriate directory on the Transfer datastore, using cp &, then running watch ls -l on the destination, if you want a progress indicator
• Re-add the storage from the vmware settings, finding it in the place you just copied it
• Power On VM, check it works, then hand back control to customer, and start a storage vMotion to relocate storage back into the correct primary datastore

All done! No messing around on the NetApp making flexclones and mounting them, cleaning them up etc. Depending on your level of risk tolerance, you could copy the VMDK back to the primary location also mounted via NFS, but we consider the small delay of the storage vMotion to be a price worth paying for peace of mind.

.. make people unable to use authentication methods that don’t involve giving you a password, that’s what!

Following on from the Gawker account hack, I have gone and changed a bunch of accounts, even though I may not have actually used a password I generated for Gawker, but it seemed prudent.

Lifehacker have a page up here which details the response..

Including this bit:

2) What if I logged in using Facebook Connect? Was my password compromised?
No. We never stored passwords of users who logged in using Facebook Connect. We have, however, disabled Facebook Connect logins temporarily.

*facepalm*

So what you’re saying is, not only are you incompetent, and let people steal your user/password database, you’ve now stopped the only way of stopping it from happening again??

Nothing pisses me off more than websites that require you to register or login to look at attachments on forums, for example. Facebook Connect (or ideally OpenID) are an awesome solution to the problem of having to create/maintain/worry about accounts on every site on the internet. I mean sure, there are idiots in marketing who love the idea of “rich user engagement” from tying them to your site with an account, but I think they severely overestimate their own importance.

.. but don’t get me started on janrain/rpx’s recent change that suggests you put your paypal username/password into HTML hosted on an insecure site so you can join the social engagement “story”. That’s just stupid.