Network glitches and corrupted VMs

I had a bit of a interesting Friday. I was so glad it was finally the weekend. Saturday we did a bunch of errands, including go visit our servers. See, we’ve been upgrading infrastructure to implement a second type of backup system. Saturday we were doing the last set of upgrades so we could install over the weekend.
Yes, we do all our own networking and racking.
12974536_10206263292444901_7498678361263518784_n
Saturday evening Steve is installing the new backup software. This is awesome backup software. It backs up the entire virtual machine. If we lose a virtual machine, we can just reload the entire thing and it will be back again.
Except while installing the software, there is a weird network glitch. Said network glitch caused the system to crash. The system crashes hard. The system crash corrupts some of the data on disk. The data on disk is our virtual machine files. Files are in read only mode and won’t fsck automatically.
We lose most of our production virtual machines.  We’re off the air.
IronyBlog
Possibly this was tragic, not ironic. I dunno, it’s been a long weekend.
We lost a bunch of production virtual machines to the disc corruption. We haven’t lost any data, but it’s taking some time to rebuild the machines and pull data from the other backup system and get it installed.
That means some of our websites and services, like tools.wordtothewise.com are down. It may mean you saw some bounces if you sent us mail over the weekend. Mail is back and we are communicating with the outside world again.
Steve’s working through our other services as fast as possible to get them back up and running.
(If massive server issues weren’t enough, one of the cats got a UTI so we’re having to pill her twice a day. Then last night managed to puke so hard she passed out briefly. Poor thing. She’s doing better this morning.)

Related Posts

What a week!

Yesterday, after 5pm, I was so happy. I was telling folks to have a great weekend. To take time off and relax. Have fun! Don’t work! Enjoy the weather!
Then someone pointed out it was only Thursday.
1331649
But! I got up this morning and got lots of happy Facebook notifications from friends about how TODAY was Friday. I was ready to have an awesome and productive day and go into the weekend with a clean todo list and a well planned next week.
Then I broke my mail client. Trying to add an attachment would crash everything. That got fixed that somewhere around noon.
So! I’ll just grab some lunch and get ready for a productive afternoon!
Then I broke finder.
12994434_10205838039782055_7428099273974075063_n
Yes, that is a picture of my 27 inch monitor with hundreds of Windows opening. I was trying to delete some of the 39,000 .jpgs from my mail client. My finger slipped on the trackpad, though, and instead of “move to trash” I clicked “show in containing folder.” Ooops. I finally crashed finder manually and it restarted and didn’t try and reopen all the windows.
OK. Fine. I’ll go to the bank and pick up mail and drop off tax (ugh, ow) payments.
On the way there, construction screwed up traffic and it took me more than 20 minutes to go 2 miles. (It’s not a safe place to walk, or I would have). On the way back, I went the Other Way. Only to discover a firetruck across 4 lanes of traffic and half a dozen cop cars showing up to a very recent accident.
Then, while writing this blog post I managed to somehow move widgets around and lose them on the wordpress editor.
Apparently I should have taken my friend’s advice and just decided today was not a work day. Because, wow, was it a mess. What all this means is I’m not going to try and blog anything substantial. I’d probably make some total boneheaded mistake and that wouldn’t be any good.
Instead, I will share the song KFOG played every Friday at 5pm (before Cumulus decided to fire everyone). Because I am really in the need of this week to be over.
Have a good weekend. Next week will be better!

Read More

Spam illustrated

Portraits of Spammers
It’s been a long week, so enjoy some art (and spam). Next week we’ll get back to discussing the many faults of Gmail. And senders. And receivers. And, well, everyone has faults. And email is Dead. Tabs killed it.

Read More

Holiday season

We’re 10 days out from Christmas, 9 days out from the end of binge-shopping-season (and 11 days out from return season). Unlike previous years, I haven’t heard of any significant delivery challenges. Most of what I’m hearing is the normal day-to-day stuff. There’s a little more of it, but nothing like in years past where ISPs melted down or giant companies got SBLed.
This is all good! This is progress and is great for senders.
Things here, and I’m pretty sure many other places are slowing down. We’re looking forward to next year, to new projects and clients, to new challenges and changes.
Blogging will probably be slow from now through the end of the year. I have stuff to talk about, but the issues are complex and I’m working on the best way to write about them. And I’m coming to the decision that writing might not be the best for certain posts.

Read More