Network glitches and corrupted VMs

I had a bit of a interesting Friday. I was so glad it was finally the weekend. Saturday we did a bunch of errands, including go visit our servers. See, we’ve been upgrading infrastructure to implement a second type of backup system. Saturday we were doing the last set of upgrades so we could install over the weekend.
Yes, we do all our own networking and racking.
12974536_10206263292444901_7498678361263518784_n
Saturday evening Steve is installing the new backup software. This is awesome backup software. It backs up the entire virtual machine. If we lose a virtual machine, we can just reload the entire thing and it will be back again.
Except while installing the software, there is a weird network glitch. Said network glitch caused the system to crash. The system crashes hard. The system crash corrupts some of the data on disk. The data on disk is our virtual machine files. Files are in read only mode and won’t fsck automatically.
We lose most of our production virtual machines.  We’re off the air.
IronyBlog
Possibly this was tragic, not ironic. I dunno, it’s been a long weekend.
We lost a bunch of production virtual machines to the disc corruption. We haven’t lost any data, but it’s taking some time to rebuild the machines and pull data from the other backup system and get it installed.
That means some of our websites and services, like tools.wordtothewise.com are down. It may mean you saw some bounces if you sent us mail over the weekend. Mail is back and we are communicating with the outside world again.
Steve’s working through our other services as fast as possible to get them back up and running.
(If massive server issues weren’t enough, one of the cats got a UTI so we’re having to pill her twice a day. Then last night managed to puke so hard she passed out briefly. Poor thing. She’s doing better this morning.)

Related Posts

Ken gets spam

Ken at Popehat gets spam offering to write a sponsored post. Ken counters with what he wants out of a sponsored post.
Blogging is going to be light this week. I’ve got a couple project deadlines to meet and most of my focus is on that. Plus, it’s the end of August and most folks are on vacation.

Read More

What a week!

Yesterday, after 5pm, I was so happy. I was telling folks to have a great weekend. To take time off and relax. Have fun! Don’t work! Enjoy the weather!
Then someone pointed out it was only Thursday.
1331649
But! I got up this morning and got lots of happy Facebook notifications from friends about how TODAY was Friday. I was ready to have an awesome and productive day and go into the weekend with a clean todo list and a well planned next week.
Then I broke my mail client. Trying to add an attachment would crash everything. That got fixed that somewhere around noon.
So! I’ll just grab some lunch and get ready for a productive afternoon!
Then I broke finder.
12994434_10205838039782055_7428099273974075063_n
Yes, that is a picture of my 27 inch monitor with hundreds of Windows opening. I was trying to delete some of the 39,000 .jpgs from my mail client. My finger slipped on the trackpad, though, and instead of “move to trash” I clicked “show in containing folder.” Ooops. I finally crashed finder manually and it restarted and didn’t try and reopen all the windows.
OK. Fine. I’ll go to the bank and pick up mail and drop off tax (ugh, ow) payments.
On the way there, construction screwed up traffic and it took me more than 20 minutes to go 2 miles. (It’s not a safe place to walk, or I would have). On the way back, I went the Other Way. Only to discover a firetruck across 4 lanes of traffic and half a dozen cop cars showing up to a very recent accident.
Then, while writing this blog post I managed to somehow move widgets around and lose them on the wordpress editor.
Apparently I should have taken my friend’s advice and just decided today was not a work day. Because, wow, was it a mess. What all this means is I’m not going to try and blog anything substantial. I’d probably make some total boneheaded mistake and that wouldn’t be any good.
Instead, I will share the song KFOG played every Friday at 5pm (before Cumulus decided to fire everyone). Because I am really in the need of this week to be over.
Have a good weekend. Next week will be better!

Read More

Unexpected break

Sorry for the unexpected break in blogging. Been dealing with some emergencies. Happy 4th to my fellow citizens. Happy late Canada day to all our northern friends. We’ll resume blogging next week.

Read More