Making Sausage
More lips and ***holes please.
Last week while I at TechEd I had a conversation with one of my Application Analysts back in the shop about a problem database from an ISV that we have been dealing with for over a month. A patch had been put in place and I was reviewing the results of some queries against the sys.dm_os_wait_stats DMV. I noticed some new issues and those yet unresolved in terms of I/O (this instance is virtual) and Full Text Index search waits.
The users were seeing an improvement – I was seeing issues internally.
That begs the question: if you see how the sausage is being made do you tell the people out in the restaurant all about it or do you keep your trap shut? If the users see an improvement and are happy with the current state of performance in their database do you continue to attempt to optimize the processes?
DBAs fall into this trap whenever we implement new monitoring or purchase and configure new third party monitoring tools – particularly when we didn’t have any similar processes in place first. You become inundated 100’s of messages:
“YOU ARE DOING THIS WRONG.”
“THIS SERVER IS HORRIBLY-CONFIGURED.”
“THIS DATBASE SHOULD COME WITH A BANJO AND OVERALLS WITH ONE STRAP MISSING.”
“DID YOU GET YOUR TRAINING IN A GRANADIAN TECHNICAL COLLEGE?”
“WALGREEN’S IS HIRING.”
“THE VAN IS IDLING OUTSIDE READY TO TAKE YOU BACK TO THE HOME.”
“HOW DID YOU MANAGE TO CONVINCE THE HUMAN RESOURCES DEPARTMENT TO HIRE YOU? BLACKMAIL?”
Relax. Take a step back and a deep break. Go get a cup of coffee. Go into the cubicle of one of the Developers and stink it up. Do whatever you need to do to calm yourself and then realize that it will all be okay. Many of these issues are benign. Many may not be, but your systems have been making sausage all along that your customers devour without hesitation. Start picking individual issues to look into and tweak the reporting responses and thresholds for your monitoring tools to fit your environment. Remember, these third-party tools are shipped with defaults that probably will not apply to your systems and you will have work to do in order to configure them to fit your environment. Over time you will end up with monitoring tailored to your systems and the 500 messages/day will be something you laugh about when you look back at it.
We see how the sausage is made on a daily basis. Understand this though: all sausage is nasty – the important thing is to make sure we do our best to keep the high fructose corn syrup, mud, cat hair, depleated rechargeable batteries, office paper, Chinese baby toys, cracked iPhones, disposable diapers, Play-Doh, coffee filters, grass clippings, kittens, Windows ME DVDs, Oprah’s discarded socks, wax really gross ingredients from reaching the plates of our customers while still giving them all the ground lips and snouts our users crave.