What did you do?  You broke TS!  I mean, it had to be you; couldn’t be me, I’m infallible!  Okay, now that we’ve got that out of the way, let me talk briefly about what went wrong, as the industrious TicketBot works to bring tickets back to our ticket site:

TicketStumbler being down for a couple days was caused by a combination of a few things:

1. Massive increases in code efficiency exposed a rather prominent bottleneck in some of our hardware that wasn’t apparent before
2. Increasing levels of disk corruption on one of our machines was becoming a serious issue, and
3. A database-related anomaly that neither myself nor 3 very skilled Database Administrators could figure out required us to accelerate our plans to migrate to a new infrastructure

Not so Fast!

As I mentioned in my previous post, before even more shit hit the fan, I had made the calculated error of creating code that was too awesome.  This awesome code had proved too much for the main database to handle so I was stuck babysitting it while adding more tickets.  While inconvenient, it wasn’t a show stopper for that particular day, but did motivate me to speed up plans to get our new “omg it’s huge” server.

Disks == Government; Corrupt!

Another one of our machines’ file systems started corrupting really often.  This required back-up restorations and occasionally caused us to have to repost stuff to the blog (it hosts the blog, by the by).  This started happening more often and, without a spare machine to move stuff to, it was strike TWO.  I can only imagine it became corrupt because I took great fun in berating this particular database machine.  Stupid MySQL, how I hate thee!  That, or the hard drives were just starting to fail, but that’s less… fun.  Usually I’d just have the drives swapped, but with two strikes and one on the horizon (I could feel it; I am a Database Whisperer, after all), it didn’t seem worth the aggravation.

Ghosts in the Machine

Okay, so two strikes.  That warranted putting in the order for the new hardware, but not shutting down TS.  Even though we didn’t have any actual tickets at the time.  I considered that only a minor inconvenience.  What came next was very… strange.  I started the process of adding back the tickets.  Mind you, this is a process that has been done hundreds of times, but this time would be different.

SELECT * FROM ticket_tmp;

For those of you who have seen SQL you’ll recognize this as a very simple query.  Basically, it says “Select every column of every row from the ticket_tmp table and give them to me[, with much haste!!!]“  The last part in brackets is usually pretty true, but suddenly the database decided that it should take at least a minute to perform a query that should take a few hundred milliseconds.  ”Okay,” said I, “something is rotten in Denmark.”  To deal with the evil Dane-related issue, I first ran some tests.  Then some more.  The numbers made no sense.  Tweaked config? Nothing.  Drop the table?  Nope.  Re-create the entire database?!  No!  Why, Danes?!  Enlist the help of people way smarter than me because certainly they can figure it out?  Baffled.

So, basically, this query and ones related to it were taking so long that I couldn’t replenish the ticket reserves of our once-great ticket site.  Dan told me tickets were pretty critical to our success and I reluctantly accepted this.  What could I have done?  Well, I could have moved the Database to another machine temporarily, but it was already slow as molasses (snails?) and I’d basically end up moving all this stuff around just to move it again in a few days.  Seemed pretty silly.  I could have temporarily inserted the tickets directly into the main table, curtailing this issue, but for various technical reasons I won’t get into that would be so much slower I’d probably die of lung cancer before we got all our tickets back.

Sometimes the right answer to a question you can’t figure out is, “Ask a different question.”  By eliminating all the variables of the current machine, the question answered itself, without days of debugging.  And now things are awesome.

Well, What Took So Long?!

I WAS COMPILING!  And this is a complicated case, Reader. A lot of ins, a lot of outs, a lot of what-have-yous, a lot of strands to keep in my head, man. Lot of strands in old Duder’s head.

Six of 12 machines have been migrated to the new hardware; only what was mandatory to get us up and running again.  The others will just have to tough it out for a while so we can get some other work done (work you can see).

Good News: New Ticket Categories!

Because I felt so monumentally bad about this entire sequence of events (which are still entirely your fault, by the way) I decided to add the following new Sport ticket categories to the site:

• NCAA Basketball (not technically new, but starts again soon!)
• MLS Cup
• NASCAR
• Boxing
• WWE
• UFC

Now, I don’t consider all of those to be sports, but I’m sure somebody does, and maybe that somebody is you.  In which case, enjoy.  And I totally recognize [category in question] as the best thing since approximately ever.  For now, you can access these sorts of events via search or by going to the sports directory listing.  When we get that blasted new site structure up, some may be more prominent. 

That’s about it!  We’re really big on transparency so any time something off-the-wall (or on-the-wall) happens that causes us downtime or other issues, rest assured you’ll hear about what happened in short order.  Sometimes it’ll be embarrassing, other times crazy.  Maybe you don’t even care what happened and that’s totally cool, too.  I enjoy reading my own writing enough to make up for it!

Extra Special Hidden Announcement!

Tricked ya!  Inspired by our good friends over at Posterous, I submit for your instant approval, our new… slogan, thing:

 Any vague references to Posterous’ own image are entirely coincidental.  This work (c)2008-Infinity, TicketStumbler Inc. and is a registered trademark of TicketStumbler Inc.

As it turns out, I was wrong.  I am human after all!

• 1200: New ticket update introduces anomalous timezone-related bug that causes many events to have the incorrect date/time.
• 1300: The error is tracked down and, influenced by the genius created by not having slept for 21 hours, I create a solution.
• 1330: The solution is implemented.
• 1350: I realize the “solution” actually had the potential to move a lot of tickets to the wrong events.  Unfortunately, once moved, there’s no way to figure out where they should go back to without making some assumptions.
• 1400: So begins the arduous task of re-initializing all ticket listings from scratch because the possibility of giving inaccurate data isn’t something to play around with.

What this means, besides further sleep deprivation for me, is that events will disappear/reappear on the site throughout the day and listings will be sparse for a little while.  All these issues should be resolved in a few hours.

The accuracy of our listings is very important to us so we make every effort to track down issues and stop them before they can spread, but once in a great while things slip by.  I apologize profusely for any inconvenience this may cause

This post rated PG (Pure Genius Injected)

It’s Tom again, you friendly neighborhood TicketStumbler developer, here to bring you lots of good news! This post could easily be thousands of words long, but for the sake of our generation, I will strive for brevity mixed with a healthy does of ambiguity.

Beginnings of Concerts

For those early adopters, you can now begin finding concerts via search here at TicketStumbler.  As of this writing, we have concerts from StubHub and TicketNetwork with more providers being added to the fold over the next day or so.  Usually this would be done faster, but silly me, I’ve gone and created code so fast that our Database cannot keep up and must be monitored until a new server can be brought online, less the current one explode — and I am not paying for that whole rack!  Concerts will take a prominent position alongside Sports and soon Theatre in the (sorta?) near future.  We’re happy to add other event types you folks want — please suggest some because for each additional million tickets I stuff into the Database, Dan gives me a cookie.  I thoroughly enjoy cookies!

I know it’s just search and just a few providers, but there’s this philosophy I live by that goes, I do what I want. As in, I want you to be able to find concert tickets a little early, before we’ve put on the finishes touches.  So go wild.  Nobody’s watching…

New Site Structure

If Dan and I could pick only one thing to hate, which is certainly impossible, it would be clutter. In the spirit of that, we will be changing our site structure over the coming days to separate Concerts and Sports on the site.  A new homepage will be added which will aggregate a sensible amount of new, upcoming, and interesting events from all of our ticket categories which you’ll be able to tab through at your leisure.  Not only will this make sure clutter doesn’t creep onto the main pages, it will also allow us to provide unique features and content on a per-category basis.

Since we’re on the topic, I might as well mention that the main blog will be split up on a per-category basis as well.  We have a serious problem here at TS:  Too damn many people want to write for us and there’s just too much important ticketing news to fit on a single blog.  With up to half a dozen posts per day on the main blog, it’s easy for stuff to get buried and that’s just not cool.  That’s why I keep things lean on this blog; for you.  More information on these changes as they develop.

Of course, by using the search you will always get results from all our events, regardless of category.  Only because we love you.

New Provider

We’re indifferent-bordering-on-ecstatic to announce we now have tickets from the following new (to TS, anyway!) provider:

• TicketNetwork

Roughly 3-6 others will be added in the near-term and as always we will continue to add high-quality, reputable, safe ticket providers as we’re able… or as they’re able to handle the traffic

Looking Ahead

We have so many new features coming up that it’s getting difficult to decide which we want right away and which can wait a little while (Psst, using the feedback forum would help us!).  After the new site structure is complete and live, work will begin on our User Dashboard feature which will allow you to do so many badass things we haven’t even come up with all of them yet, lest we overload our threshold for badassery.

To start, though, you’ll be given the ability to register on the site (using existing accounts from AIM, Google, Yahoo and many others — because who needs another login to remember?) which will allow you to setup alerts and feeds for… well, almost anything.  Get alerts when new tickets are available for events, when specific search results are updated, whatever.  Don’t feel like coming to site every day and filtering through tickets for an event you just can’t miss? Good, I’m sick of seeing you around here anyway; you should just setup filters in your account and we’ll alert you of new tickets which fall within them (for that event your significant other will kill you for not getting tickets to!)

Hey, this is just the beginning.  We’ve got a lot more lined up which we’ll be announcing as features get close to release.  We also have one feature which will, quite literally, cause our competitors to burst into flame due to jealousy. We’ll be mailing out fire extinguishers soon to help avoid any loss of life, although we cannot stop the inevitable loss of hope and feelings of melancholy which will result.

As a final note, I will also start being more active around here.  I’ll be providing weekly change logs (non-geek: “updates”) regarding minor new features, bug fixes, changes, etc.  I’ve also got some posts lined up on other rant-like topics.  Lets be honest here: This the best day of your life because you’ll get to read more of my super awesome writing.  Like, whoa.

Looking Behind

Ever look behind you? There’s scary stuff back there.  That’s why we don’t have eyes in the back of our heads.  Scientific fact.  Anyhoo… what was I saying? Oh, right.

It’s been a really long road just to get to… what?  Some concerts and a couple new site sections?  Some of you may be wondering why that is and I’ll be covering that topic in another post.  For now, I want to ease your collective minds that such profoundly long release cycles are now a thing of the past and we can all move forward amicably.  Depending on interest, we can have theatre tickets up in two shakes of a lamb’s tail, for example.  Well, after I put out the fires on the servers anyway.

That being said, lets not get ahead of ourselves, okay?  We can’t all be like Posterous. They get better every day. A couple of cocky over-achievers if you ask me.  Dan suggested we should also get better every day once, but I threw a Molotov cocktail at his face and that pretty much ended that discussion.  I’m going to shoot for every few days instead.  Although, if you’re really motivated for the every-day thing, we are hiring.  And by hiring I mean I will bestow upon you the honor of doing some of my work for free; there’s plenty to go around.

… Please?  I’ll wash your car?  Or dog?  Or… you?

So I just type anything I want and it shows up here?

Well, that’s pretty much all I’ve got for now.  Oh, wait, here are some big numbers.  Big numbers == impressive, unless it’s my credit card statement — these will probably increase by 50% within the next few days.  That’s nearly 1/2 of 100%!  Stick that in your pipe and smoke it, other dudes!!! (dudettes?)

Now Serving:  8,191,150 Tickets representing 2,151,170 listings to 20,562 Events!

You know what they say about men with big numbers.  They work at TicketStumbler!  ZZIIINNNGGGG!!!!