OH THE HORROR! (and new tickets!)

What did you do?  You broke TS!  I mean, it had to be you; couldn’t be me, I’m infallible!  Okay, now that we’ve got that out of the way, let me talk briefly about what went wrong, as the industrious TicketBot works to bring tickets back to our ticket site:

TicketStumbler being down for a couple days was caused by a combination of a few things:

1. Massive increases in code efficiency exposed a rather prominent bottleneck in some of our hardware that wasn’t apparent before
2. Increasing levels of disk corruption on one of our machines was becoming a serious issue, and
3. A database-related anomaly that neither myself nor 3 very skilled Database Administrators could figure out required us to accelerate our plans to migrate to a new infrastructure

Not so Fast!

As I mentioned in my previous post, before even more shit hit the fan, I had made the calculated error of creating code that was too awesome.  This awesome code had proved too much for the main database to handle so I was stuck babysitting it while adding more tickets.  While inconvenient, it wasn’t a show stopper for that particular day, but did motivate me to speed up plans to get our new “omg it’s huge” server.

Disks == Government; Corrupt!

Another one of our machines’ file systems started corrupting really often.  This required back-up restorations and occasionally caused us to have to repost stuff to the blog (it hosts the blog, by the by).  This started happening more often and, without a spare machine to move stuff to, it was strike TWO.  I can only imagine it became corrupt because I took great fun in berating this particular database machine.  Stupid MySQL, how I hate thee!  That, or the hard drives were just starting to fail, but that’s less… fun.  Usually I’d just have the drives swapped, but with two strikes and one on the horizon (I could feel it; I am a Database Whisperer, after all), it didn’t seem worth the aggravation.

Ghosts in the Machine

Okay, so two strikes.  That warranted putting in the order for the new hardware, but not shutting down TS.  Even though we didn’t have any actual tickets at the time.  I considered that only a minor inconvenience.  What came next was very… strange.  I started the process of adding back the tickets.  Mind you, this is a process that has been done hundreds of times, but this time would be different.

SELECT * FROM ticket_tmp;

For those of you who have seen SQL you’ll recognize this as a very simple query.  Basically, it says “Select every column of every row from the ticket_tmp table and give them to me[, with much haste!!!]“  The last part in brackets is usually pretty true, but suddenly the database decided that it should take at least a minute to perform a query that should take a few hundred milliseconds.  ”Okay,” said I, “something is rotten in Denmark.”  To deal with the evil Dane-related issue, I first ran some tests.  Then some more.  The numbers made no sense.  Tweaked config? Nothing.  Drop the table?  Nope.  Re-create the entire database?!  No!  Why, Danes?!  Enlist the help of people way smarter than me because certainly they can figure it out?  Baffled.

So, basically, this query and ones related to it were taking so long that I couldn’t replenish the ticket reserves of our once-great ticket site.  Dan told me tickets were pretty critical to our success and I reluctantly accepted this.  What could I have done?  Well, I could have moved the Database to another machine temporarily, but it was already slow as molasses (snails?) and I’d basically end up moving all this stuff around just to move it again in a few days.  Seemed pretty silly.  I could have temporarily inserted the tickets directly into the main table, curtailing this issue, but for various technical reasons I won’t get into that would be so much slower I’d probably die of lung cancer before we got all our tickets back.

Sometimes the right answer to a question you can’t figure out is, “Ask a different question.”  By eliminating all the variables of the current machine, the question answered itself, without days of debugging.  And now things are awesome.

Well, What Took So Long?!

I WAS COMPILING!  And this is a complicated case, Reader. A lot of ins, a lot of outs, a lot of what-have-yous, a lot of strands to keep in my head, man. Lot of strands in old Duder’s head.

Six of 12 machines have been migrated to the new hardware; only what was mandatory to get us up and running again.  The others will just have to tough it out for a while so we can get some other work done (work you can see).

Good News: New Ticket Categories!

Because I felt so monumentally bad about this entire sequence of events (which are still entirely your fault, by the way) I decided to add the following new Sport ticket categories to the site:

• NCAA Basketball (not technically new, but starts again soon!)
• MLS Cup
• NASCAR
• Boxing
• WWE
• UFC

Now, I don’t consider all of those to be sports, but I’m sure somebody does, and maybe that somebody is you.  In which case, enjoy.  And I totally recognize [category in question] as the best thing since approximately ever.  For now, you can access these sorts of events via search or by going to the sports directory listing.  When we get that blasted new site structure up, some may be more prominent. 

That’s about it!  We’re really big on transparency so any time something off-the-wall (or on-the-wall) happens that causes us downtime or other issues, rest assured you’ll hear about what happened in short order.  Sometimes it’ll be embarrassing, other times crazy.  Maybe you don’t even care what happened and that’s totally cool, too.  I enjoy reading my own writing enough to make up for it!

Extra Special Hidden Announcement!

Tricked ya!  Inspired by our good friends over at Posterous, I submit for your instant approval, our new… slogan, thing:

 Any vague references to Posterous’ own image are entirely coincidental.  This work (c)2008-Infinity, TicketStumbler Inc. and is a registered trademark of TicketStumbler Inc.