Caffeinated-Geek.com

Received a complaint from a co-worker today – the long and short of it is his battery backup (UPS) wasn’t working properly. Upon suggesting that I try testing/replacing the batteries, I was told the batteries were new. I mentioned that since I was working on a customer-affecting issue, I’ll take a look shortly, and asked him to please email me with the battery date on the sticker affixed to the case of the UPS – and that will help determine our plan of action. His response was that “that’s not my job” and to “go look at it yourself”.

I figure he was just having a bad morning, but it brings up the topic of what you can do to help IT help you with your problems (also known as “how to NOT piss off IT”). Without getting into too many intricate details, my list is:

• Be patient. I realize your issue is important, but troubleshooting problems (be they yours or someone who is in queue ahead of you) takes time. Please don’t assume because something isn’t done immediately that you need to follow up by phone or in person – I work on tickets according to priority, then submission date. If you choose to call, you’re probably going to end up in voicemail and getting a call back when I’ve cleaned out my ticket queue. If you come to my desk, you’re likely going to end standing there waiting until I wrap up my current ticket or call.
• Set realistic priorities. At any point throughout the day, I probably have a minimum of 5 – 10 tickets open in my own queue, often many more; that means there’s only about a 1-in-10 chance that yours really is more important than everyone else’s. Continually marking your low-priority issues as an “emergency” will not get them fixed faster, but may get your true high-priority issue you open later pushed to the bottom of the queue.
• Don’t lie. If you changed something that may be related to your problem, man up. If you’re consistently going to let me waste my time troubleshooting instead of coming clean and providing me the whole story, then when the time comes that you have a real issue it’s going to take longer to get resolved since I’ll be spending my time looking to see what you screwed up but won’t admit to.
• Give details and note error messages. Provide usernames, email addresses, and callback numbers. It’s much faster for you to provide the information I need to troubleshoot than it is for me to go back and forth trying to squeeze information from you or wading through a metric ton of server logs. I’m not going to troubleshoot in the dark — if you send a ticket saying “email is down”, I’ll respond with an equally vague message saying “it’s working fine for me”. Doing this one step alone could mean the difference between having the problem persist for a few minutes or a few hours.
• Don’t argue, clarify. If you think I’m wrong about something, ask for clarification or explain that you thought it worked differently, but don’t simply start an argument. Not to be rude, but you called me for help and I’ve been dealing with issues like this for a long time. Almost every time someone wants to argue, it comes down to them not completely understanding the intricacies of protocols and services such as BGP, ATM, PPP, DNS, and SMTP. If you know I’m wrong, explain why and I’ll listen and admit it if so — and we can continue with getting your problem fixed. If you feel the need to have an argumentative conversation, please don’t waste my time – there are plenty of Internet forums out there for you to troll.
• Don’t play the blame game. Bickering about who is at fault for your document getting deleted, your workstation crashing, or your email bouncing is not going to resolve the issue. Technology breaks, mistakes happen, life goes on. Deal with SLA’s per your contract, but AFTER the service-affecting issue is resolved; don’t attack the person on the other end of the phone when it comes to settling the dispute.

Anything else I’m missing?

Complaints, Systems Admin headdesk, lusers

So, a thread about stupid user passwords recently came up on a group that I frequent, and I thought I’d post this here.

We store customer information in MySQL, and have to keep a cleartext password for PPP CHAP authentication. A while back, I did some querying to see just how terrible our users’ passwords were.

Here were some of the more interesting/amusing results (remember, in SQL quotes surround literal strings and “%” is a wildcard):

SELECT COUNT(*) FROM customers: 32112

SELECT COUNT(*) FROM customers WHERE password = “password”: 151

SELECT COUNT(*) FROM customers WHERE password = username: 660

SELECT COUNT(*) FROM customers WHERE password LIKE “123%”: 364

SELECT COUNT(*) FROM customers WHERE password LIKE “%321″: 44

SELECT COUNT(*) FROM customers WHERE password LIKE “qwerty%”: 8

SELECT COUNT(*) FROM customers WHERE password LIKE “asdf%”: 11

SELECT COUNT(*) FROM customers WHERE password = “********”: 16

SELECT COUNT(*) FROM customers WHERE LENGTH(password) <= 4: 5151

…and I thought our users were doing surprisingly well — until I executed the last query.

Humor, Systems Admin headdesk, password

Got a spam complaint about one of our customers today. Looked up the call-from number in a public database and recognized the name – as a spammer whose account I nuked from one of our ISPs back in 2007. He signed up for a different account on a different one of our ISPs a couple of days ago. Spamming from dial-up seems kinda lame, but whatever floats his boat. I still have a huge file on this guy with loads of his personal and business info… maybe I take these things too personally sometimes.

Oh well, back to business:

• Nuking spammer’s account: Good clean American fun.
• Blocking his phone number on our dialups: More fun, more satisfying, and makes me feel manly.
• Looking up his info, getting his apartment complex name from Google Street View, calling his building manager and convincing them to give up our spammer’s cell phone number, then calling him to PERSONALLY let him know that I removed his account and banned his phone numbers: Priceless.

He told me it was illegal for me to call him. I told him that I appreciate his sub-par legal advice and advised him that he might want to read up on fraud and felony computer crime statutes in our state. Asked him to have a great afternoon and hung up.

Thank you for playing, don’t let the door hit you on the way out.

Humor, Systems Admin bofh, spam

There are some days that I get completely fed up with the IT career path I’ve picked for myself. Then, there are some where I love what I do. I thoroughly enjoy when, even though there is a problem, everything gets worked through to completion properly and I can make someone happy – especially when it’s my parents.

Mom and Dad finally decided it’s time to upgrade from dial-up to DSL. Both of them had high-speed access at work (in fact, I set up the DSL at Dad’s office), but there’s been nothing faster than dial-up in that house since they moved in — and that was finally about to change.

When I lived with them, I had dial-up on a dedicated second pair coming into the MPOE/NID that was cross-connected via CAT3 that we pulled around the house and drilled into a wall/interconnected in a modular jack in my old room. At the time for where we lived, dial-up at 33.6 was definitely better than any neighbors could ever ask for. When I moved out, there was no real need for the second phone line for net access anymore, so it was dropped.

Unfortunately, dial-up access on the primary line was horribly slow and unreliable – I always chalked it up to the inside wiring, and threw some inline microfilters on all extensions, which helped a bit. It was never too much of a concern for them, since they didn’t use the connection a lot, and they didn’t want to be a bother about it – Mom always says, “don’t worry about it, it’s not important” and Dad jokes about not being able to type quickly enough for faster internet access anyway.

Today was their DSL due date. I come out to help hook it up, disable automated dialing of the old connection, and to make sure it worked well. Unfortunately, it was working much slower than expected, especially for a 1.5mbit product. Assuming it was the inside wiring once again, I went around replacing microfilters through the house and checked the inside jack’s wiring – but to no effect. What a killjoy; 1.5mbit down/1.0mbit up DSL that works at 256kbit down and 128kbit up, which is slower than the lowest speed offering from the phone company.

Not in the mood to be beaten by the house wiring again, I head out to the car to grab my test set and other tools. I take out a box knife and run it along the seam of the NID, removing a nice layer of paint that’s been curing in the sun over it for the past 6 years. Screwdriver in one hand, cordless phone in another, and lineman’s handset hanging to my side, I make entry into the NID, bound and determined to make the horrid copper bundle inside cower in fear at my geekiness.

I note the wiring job from over 10 years ago when we ran the CAT3 cable from the second line into the house, and admire how it has held up. I again start to wonder why the original wiring was done with an interesting 7-pair bundle running to two modular jacks, and who in the heck decided to ignore the standards for color coding telecom wiring by hooking up each jack to random color pairs. Time to get down to business – off comes the house wiring, when I notice one of the joys of copper – a familiar green foe. Oxidation, my old enemy – at last we meet again. I clamp the test set to the terminal posts and all is well until I brush one of the insulated wires and unleash a crackle of fury out of the handset speaker.

Off with the terminal posts and wires; out comes the metal brush. Goodbye, gunky green evilness. Hook it all back up and test with the lineman’s handset again, and the quality sounds great. Plug the home wiring interconnect back in and head inside to watch the DSL sync up at full speed. Success!

Head back outside, make pelvic thrusting motion towards the NID to show my superiority, and close it up.

I’m happy. Mom’s ecstatic. Dad still types slowly, but is happy too. I am rewarded with unending gratitude and food to take home for dinner. It has been a good day.

Network Admin, Systems Admin

I love three day weekends. Three consecutive 24 hour periods of hanging out with friends and family, finishing projects, and all around laziness.

A wicked summers-almost-over barbecue with the whole family, working on my stereo install in my car, and lounging around at home were on my agenda – but not dealing with people who forgot that this was the end of the month, and that they needed to pay their bill to us (or they’d get shut off).

Background: I’m a systems and network admin for a wholesale ISP. We provide dial-up, DSL, hosting, etc. Some of our wholesale customers use their own RADIUS system for authentication, some use a managed system on our side. It’s in violation of our contract with the wholesale ISP to activate accounts/tinker with the accounting functions directly for a subscriber in our managed system, and it’s impossible for me to activate an account on a system that they manage.

There’s something about a three day weekend when the calendar month rolls over that makes our wholesale customers forget to do little things like paying their bills. I can’t take it out on the poor technical people who have to call me; they’re usually just reacting to customers yelling at them. It’s their management, bookkeepers, accounts payable, whoever is responsible in their organization that has dropped the ball. What irks me the most is that we notify people if they haven’t paid NUMEROUS times before shutoff — and it doesn’t help. And that’s what causes my cellphone to ring non-stop this weekend.

Since the ISP’s tech folks don’t usually know that their management has neglected our invoice, it simply looks like a massive technical issue as their retail customers can’t log on, and they call our emergency outage paging system, which patches them through to me – which is when I get to inform them that their boss never paid us. Most of them, I can turn back on right away and have them take care of it on the next business day. There are others that are persistently late, and that I need confirmed payment from to turn back on. Of course, the person who handles that is out of town for the holiday, too. Great.

Better than the wholesale calls, though, are the retail customers — who aren’t supposed to be calling us at all. They usually come across the NOC phone number by stumbling across it in WHOIS, or by talking to the phone company (who gives our contact info as the service provider for their DSL, since they’re unaware of our wholesale program), or when given it by the wholesale partner. Note that a wholesale partner doing the latter is grounds to have them stuffed into a cannon and shot at the Earth’s sun. Oh, and I can’t forget to mention that part of the telephone IVR greeting says that if you’re an end-user, to not use the emergency paging system. They never listen and proceed to the paging system anyway.

The fun really begins when they get connected with me; the end users want to argue with me about how they are consistently on time with payments, and this is unfair, and how they’re going to go to another service provider — even after I’ve explained that I’m at a wholesale/upstream provider level and have no access to the accounting and user login functions for their service provider. Yes, they might be the perfect customer or they may have been turned down mistakenly, but it doesn’t change the fact that I cannot do anything for them. Yet, somehow, I’m expected to turn them back on, offer a credit for an account that doesn’t belong to us, and publish a three page letter to the local newspaper apologizing for the actions of one of our customer.

I’ll get right on that first thing tomorrow.

This all brings to mind an old statement I first heard several years ago said by a co-worker to a member of the sales department:

“Failure to plan on your part does not constitute an emergency on mine.”

Complaints, Network Admin, Systems Admin headdesk, lusers