Time for a bit of a rant.
I came in to work early today, hoping to get in some project time. No less than 10 minutes after my arrival, I’m noticing our bandwidth is seriously pegged. Pegged, as in one Fast Ethernet connection and one DS3 completely saturated. Some investigation showed that a site on one of our hosting servers had apparently received enormous press coverage, and we had thousands of users browsing through an image gallery with hundreds of photos on each page, and of course the “thumbnails” created via HTML. Ugh, IMG WIDTH and HEIGHT attributes do NOT make thumbnails people. Add to that video downloads of a 400MB AVI from poorly configured streaming clients, and I was not happy. Anyway…
Now that I’ve located the geyser, it’s time to find out why it’s dragging everything down. While attempting to stop the problem, I notice the culprit — a web server upgrade killed the bandwidth throttling module. Shit. Well, there’s more than one way to skin a domesticated feline — time to filter on the network level, or so I thought. You see, the version of IOS in use has a lovely bug on the Cisco 7200 platform which doesn’t allow me to filter outbound traffic on any interfaces. 30 minutes into attempting to beat my ACLs into submission, I jump 5 feet into the air as my cellphone rings. Maybe I should lay off the coffee.
Time to talk to my favorite customer. For 45 minutes. About the issue I’m attempting to fix. I let him know that I was attempting to fix the problem at that very moment, and that I’d be more than happy to call him back as soon as I was able to finish working on it, but that my hands were tied while he was complaining and throwing random questions of a questionably technical nature at me. Then, like magic, one of the hosting servers crash. Unfortunately, it’s not hosting the site that’s hammering our bandwidth. I can’t win today.
I ended up forcibly terminating the call with the customer, as they refused to let me defer their conversation to fix issues affecting thousands of people. I’m sure they’ll call back to complain. Of course, for him to understand, despite being told several times, that I’m unable to be on my phone at my desk and in the data center at the same time would require just a tiny fraction of logic. My theory is that he was in the john while the logic handouts happened.
How I love feeling helpless during service affecting conditions due to one subscriber’s lack of a logical thought process. I can’t wait to see where the rest of the day goes, as it’s only 9:05 am.
Ouch.
Complaints, Systems Admin
headdesk, lusers
