Network glitches and corrupted VMs

I had a bit of a interesting Friday. I was so glad it was finally the weekend. Saturday we did a bunch of errands, including go visit our servers. See, we’ve been upgrading infrastructure to implement a second type of backup system. Saturday we were doing the last set of upgrades so we could install over the weekend.
Yes, we do all our own networking and racking.
12974536_10206263292444901_7498678361263518784_n
Saturday evening Steve is installing the new backup software. This is awesome backup software. It backs up the entire virtual machine. If we lose a virtual machine, we can just reload the entire thing and it will be back again.
Except while installing the software, there is a weird network glitch. Said network glitch caused the system to crash. The system crashes hard. The system crash corrupts some of the data on disk. The data on disk is our virtual machine files. Files are in read only mode and won’t fsck automatically.
We lose most of our production virtual machines.  We’re off the air.
IronyBlog
Possibly this was tragic, not ironic. I dunno, it’s been a long weekend.
We lost a bunch of production virtual machines to the disc corruption. We haven’t lost any data, but it’s taking some time to rebuild the machines and pull data from the other backup system and get it installed.
That means some of our websites and services, like tools.wordtothewise.com are down. It may mean you saw some bounces if you sent us mail over the weekend. Mail is back and we are communicating with the outside world again.
Steve’s working through our other services as fast as possible to get them back up and running.
(If massive server issues weren’t enough, one of the cats got a UTI so we’re having to pill her twice a day. Then last night managed to puke so hard she passed out briefly. Poor thing. She’s doing better this morning.)

Related Posts

Lavabit shuts down

Lavabit is a secure mail system. Today their CEO announced he was shutting down the service immediately.

Read More

Unexpected break

Sorry for the unexpected break in blogging. Been dealing with some emergencies. Happy 4th to my fellow citizens. Happy late Canada day to all our northern friends. We’ll resume blogging next week.

Read More

Holiday season

We’re 10 days out from Christmas, 9 days out from the end of binge-shopping-season (and 11 days out from return season). Unlike previous years, I haven’t heard of any significant delivery challenges. Most of what I’m hearing is the normal day-to-day stuff. There’s a little more of it, but nothing like in years past where ISPs melted down or giant companies got SBLed.
This is all good! This is progress and is great for senders.
Things here, and I’m pretty sure many other places are slowing down. We’re looking forward to next year, to new projects and clients, to new challenges and changes.
Blogging will probably be slow from now through the end of the year. I have stuff to talk about, but the issues are complex and I’m working on the best way to write about them. And I’m coming to the decision that writing might not be the best for certain posts.

Read More