Making Sausage
More lips and ***holes please.
Last week while I at TechEd I had a conversation with one of my Application Analysts back in the shop about a problem database from an ISV that we have been dealing with for over a month. A patch had been put in place and I was reviewing the results of some queries against the sys.dm_os_wait_stats DMV. I noticed some new issues and those yet unresolved in terms of I/O (this instance is virtual) and Full Text Index search waits.
The users were seeing an improvement – I was seeing issues internally.
That begs the question: if you see how the sausage is being made do you tell the people out in the restaurant all about it or do you keep your trap shut? If the users see an improvement and are happy with the current state of performance in their database do you continue to attempt to optimize the processes?
DBAs fall into this trap whenever we implement new monitoring or purchase and configure new third party monitoring tools – particularly when we didn’t have any similar processes in place first. You become inundated 100’s of messages:
“YOU ARE DOING THIS WRONG.”
“THIS SERVER IS HORRIBLY-CONFIGURED.”
“THIS DATBASE SHOULD COME WITH A BANJO AND OVERALLS WITH ONE STRAP MISSING.”
“DID YOU GET YOUR TRAINING IN A GRANADIAN TECHNICAL COLLEGE?”
“WALGREEN’S IS HIRING.”
“THE VAN IS IDLING OUTSIDE READY TO TAKE YOU BACK TO THE HOME.”
“HOW DID YOU MANAGE TO CONVINCE THE HUMAN RESOURCES DEPARTMENT TO HIRE YOU? BLACKMAIL?”
Relax. Take a step back and a deep break. Go get a cup of coffee. Go into the cubicle of one of the Developers and stink it up. Do whatever you need to do to calm yourself and then realize that it will all be okay. Many of these issues are benign. Many may not be, but your systems have been making sausage all along that your customers devour without hesitation. Start picking individual issues to look into and tweak the reporting responses and thresholds for your monitoring tools to fit your environment. Remember, these third-party tools are shipped with defaults that probably will not apply to your systems and you will have work to do in order to configure them to fit your environment. Over time you will end up with monitoring tailored to your systems and the 500 messages/day will be something you laugh about when you look back at it.
We see how the sausage is made on a daily basis. Understand this though: all sausage is nasty – the important thing is to make sure we do our best to keep the high fructose corn syrup, mud, cat hair, depleated rechargeable batteries, office paper, Chinese baby toys, cracked iPhones, disposable diapers, Play-Doh, coffee filters, grass clippings, kittens, Windows ME DVDs, Oprah’s discarded socks, wax really gross ingredients from reaching the plates of our customers while still giving them all the ground lips and snouts our users crave.
Very nice and I agree 100%. It’s like the quote from ‘House’:”I don’t do full body scans because you will always find an anomaly or two”. There will always be ‘something’ in your system that looks suspicious.
I completely agree. I guarantee that if you don’t address performance problems that you see, even though users are happy, it will pile up over time and come back to bite you. Now I’m off to stink up a developer’s cube!
Nice writeup. I like the approach of let’s look at the top 5 worst things and work the list down from there. I find that typically the top 1 or 2 things create the biggest improvements and the rest seems like no big deal.
Shadowing the users is always an interesting activity. You will learn a lot seeing how people use the apps on the databases you support. Makes you realize things are not as bad as they seem.
Great post. SCOM has the tendancy to bust my chops every minute of every day. Its really the SCOM that cried wolf. I have found myself multiple times going back through the emails after the fact and seeing an alert 6 hours prior to me hearing about it from users. oops.
Problem is the really important stuff like granting a developer dbo on accident doesn’t even get alerted. So I’m with you, don’t spend too much of your day washing the lips and cheeks and watch out for the Oprah socks.