The Art of Being a Successful DBA – Keeping Customers Happy, Part 2

Introduction
In today’s business environment, being a successful DBA is more than just being technical. If you want to excel in this profession, you must be seen as someone who understands both the business and technical aspects of the applications you support.

In last week’s post, I started providing a general set of recommendations on keeping your customers happy. Today, I’ll continue our discussion on “The Art of Being a Successful DBA” with my remaining recommendations for customer satisfaction.

Application Design Reviews
Yet another activity that I firmly believe in.  Just call me “Mr. Procedure”…  I’ve just seen way too many things go wrong in 20 years in this profession to not take a proactive problem prevention approach to the development process.   The bigger the application being developed, the more proactive you need to be.

I used to be a “high priced industry consultant” that customers would pay a fairly substantial rate for.  As a result, I would be brought in to solve the “tough problems”.   What I found was that if these customers WOULD HAVE FOLLOWED ANY TYPE OF DESIGN REVIEW PROCESS THE MAJORITY OF THESE PROBLEMS WOULD *NOT* HAVE OCCURRED.  Ok, now I’m yelling.

I was once asked to be onsite for a huge application turnover for a local business.  I had about 4 hours notice to drive out to a firm and help the other consulting company when they “flipped the switch” on the new application.   Although this consultant group was from the largest technical firm in our industry, documentation (and application design best practices) was dismal at best.  I just had a very uneasy feeling.  This application was BIG.  I could also tell my fellow onsite DBA was not very experienced and very overwhelmed.  He stated that most of the DBA work was done offsite.

Well, when “they flipped the switch” – “the lights went out”.    Nothing, and I mean nothing, wanted to run.   I looked at the other consulting DBA, his eyes were as big as pie plates.  He looked at me, said “I quit” and began to pack his equipment up.   I whispered to him “don’t ruin your career over this, let’s take a look first”.

Sky-high CPU was the culprit.  The next step was to look at some diagnostic information (we’ll be covering this in-depth in upcoming blogs).  Reviewing the SQL provided in a SQL Trace dump, I noticed nothing, and I mean nothing, was using bind variables.  This application also had dozens of batch jobs that processed around the clock.  The jobs were looping and hard parsing, looping and hard parsing – statement after statement after statement.

I thought to myself “this may not be another 36 hour, around-the-clock gig”.  The database went down, the cursor sharing parameter was changed, and the database brought up.  It was like we switched to the Energizer Bunny.  The application just ran.  I thought “wow, I never knew hard vs soft parsing made that much of a difference”.    The culprit was that this very blue consulting organization used a new fangled code generator that didn’t generate bind variables.

I have been involved in dozens of cases like these.  When I asked if anyone went through a performance review during the design process, I received blank stares.  The moral of this, and many other horror stories, is that they *ALL* could have been prevented if any type of design review process was followed.

The goal of the design review process is to identify and address application design, process flow, program logic and SQL statement problems early in the development life cycle. Identifying these issues early in the development life cycle allows them to be more easily addressed than if they were to be identified during later stages.  That last statement bears repeating.   Like any other issue you face, no matter what it is (personal, professional, whatever), the sooner you identify it, the easier it is to correct.

Over the years, I have come up with a set of design review meetings that any DBA unit can follow.  These recommendations are not intended to coerce readers into using the application design review meeting examples verbatim but to emphasize the importance of a structured approach to the design review process. The seemingly endless combinations of application architectures and software products used for application development may require the creation of a fully customized design review process for each application development project. The overall goal of design review meetings is to ensure the involvement of technical support units during the application’s design and implementation. When design issues are addressed early in the development life cycle, problems are minimized and the migration from test to production is more easily accomplished.

DBA Report Cards and the 360-Degree Review Process
Effective measurements are required to judge the success of any activity. The quality of support the DBA unit provides should be reviewed on a regular basis. A DBA report card allows business and application development units to provide feedback on DBA support activities.

The report card will allow them to measure how well they feel you are meeting your Service Level Agreements.  Here are a few sample questions to start you on your way:

• How would you rate the turnaround times for DBA unit work requests?
• How would you evaluate the DBA unit’s responsiveness to questions?
• How would you evaluate the DBA unit’s responsiveness to requests for assistance?
• Please rank the quality of communications the DBA unit provides.
• Please rank the overall quality of work the DBA unit provides.

All questions are ranked from 1 to 5 with 5 being the highest. You can also allow the respondents to rank the importance of each question from 1 to 5 with 5 being the highest.

A short survey should be attached to the DBA report card to gather additional information that can be used to improve the quality of support the DBA unit provides to their customers:

• What are your top three technical challenges that you face?
• What are the top three non-technical challenges?
• Please list your current priorities. Rank them in order of importance.
• List the most important services the DBA unit provides. Rank them in order of importance.
• What support services does the DBA unit do a good job of providing?
• What support services should the DBA unit improve upon?
• What additional services would you like the DBA unit to provide?

Meetings can be held with the respondents to discuss their reviews. DBA team members participating in the reviews must be prepared to respond to criticism in a professional manner. But just as its title describes, the 360-degree review process also allows support units to provide feedback on their customer’s support requests and work activities. The 360-degree review process provides important feedback to both support units and their customers. Once again, your customers may not know that some of their expectations are unachievable until you tell them the reasons why.

Corrective Action Reports
Databases are challenging to administer. As much as we would like to prevent mistakes, we do make them. That’s one of the benefits of being human; we learn from our mistakes. During my 20 years in this field, I’ve seen all kinds of mistakes made – from little ones to catastrophic. I’ve also made my fair share of them.

The worst one I ever experienced is when we brought a technician in from a third-party disk vendor to format what we thought was to be a single disk in a huge disk array. We just purchased the arrays and were in the process of assuming support responsibilities from the vendor. The vendor assisted us in the initial setup and data migration from our old storage devices to their new whiz-bang storage system.

The array stored data from dozens of Oracle databases. After 20 minutes of showing our folks how to set the necessary switches and enter the appropriate values at the prompts, the technician hit the enter key with much flair and bravado.

I assumed we would see the one activity light activate for the disk being formatted. Instead, I saw a whole wall of lights begin flashing and twinkling. I thought to myself “Hmmm, this can’t be good…”. I glanced at the technician and the look of horror on his face confirmed my suspicions. He then reinforced my conclusions when he stammered, “I think I just formatted the entire array.” I looked at the DBA next to me and said “Looks like we are in for a looong night.”

I continue to be amazed at the rapid improvements in hardware redundancy and reliability. Hardware platforms now have the ability to recognize individual component failures, bypass them, produce diagnostic information and “phone home” to report the problem to the manufacturer. But hardware components will fail. Just as the sun comes up every morning, hardware components will fail, operating systems will get tied up in knots and database bugs will manifest themselves. Such is life for a computer technician.

The key is to document everything that happened during the time period when the problem occurred. Documenting provides you with the information you need to prevent the problem from occurring again and your customers with information on exactly what happened. Its my experience that the more informed a customer is about the problem and the steps your unit is taking to prevent its reoccurrence, the better they feel.

The corrective action report should contain:

• A detailed description of the error that caused the problem.
• The customer impact the problem caused.
• A timeline of the activities that were performed.
• The steps that were performed to correct the problem.
• Mitigating factors that contributed to or exacerbated the problem’s impact.
• The steps that will be taken to prevent the problem from occurring again. Include who is responsible for the activity and a date the activity will be completed.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery