One of the common challenges that all technicians face, no matter what area of IT they work in, is the absolute attention to detail our profession demands. Switch a couple of characters in a script, forget to set your SID, set the wrong flag at the wrong time and the end result usually isn’t very pretty. Many commands we issue on a regular basis are destructive by their very nature.

If you never make mistakes, send me a resume. I’m always looking for a “Patron Saint of IT” here at Remote DBA Experts. It will also save us on travel costs because I’m sure you’ll be able to spread your wings and fly here on your own.

Then there’s the software glitches. The problems that pop up out of the blue and make you go:

“WHAT THE? – How did THAT happen? I’ve done this 317 times in a row and it worked every time.”

For you math majors, here’s my calculation for one of the Foot Principles of IT Support:

CLOSER YOU ARE TO PRODUCTION TURNOVER

+ THE GREATER THE VISIBILITY OF THE PROJECT

= THE MORE LIKELY A PREVIOUSLY UNKNOWN SOFTWARE GLITCH WILL OCCUR

I don’t care what software you are using, you will run into the “only occurs on this release, on this version of the operating system, using this particular feature on the third Tuesday of the sixth month when it’s cloudy outside” BUG. Be sure to expect management to stop by and ask “well, why didn’t you test this on the third Tuesday of the sixth month when it was cloudy outside?”

Finally, there are the hardware problems that we have all come so accustomed to and fond of.   It’s that triple redundant, super fault tolerant, titanium based, twice-the-price component that you just got management to sign off on a few months ago.    The one that your hardware vendor assured you would solve all of your performance and availability problems.   The component they described as “state-of the art and self healing”.  Fixes itself, they said.

The one that forced you to create that 15 page justification document with all the pretty charts and graphs stating that this piece of hardware will still be running in the year 2040.   The component that just decided to flake out and take down your entire online, 10-thousand dollar a minute web-based ordering system.  You then call the vendor and they say “yeah, we’ve heard about that happening occasionally – we thought we got that fixed.”

Root Cause Corrective Action Reports

A customer that is affected by an application outage or slowdown needs to have a firm understating on  what caused the problem, the activities performed to correct the problem and the action items that will be undertaken to mitigate or prevent the problem from occurring again.

The Root Cause Corrective Action Document provides information on the underlying causal factors that generated the problem and a timeline of events that occurred during the problem event. This ensures that all problems are properly analyzed and that all steps are taken to prevent future occurrences.  This is a key component of our problem resolution strategy in addition to obtaining customer feedback on the quality of our problem resolution capabilities.

I can’t stress the importance of using some form of problem notification document.   A customer that is unsure of “what happened” is going to an unhappy customer.   Giving your customer a clear picture of the problem event and the steps you will take to prevent the problem’s reoccurrence  shows them that the quality of their environments is IMPORTANT TO YOU and you do NOT TAKE PROBLEMS LIGHTLY.    The time you spend crafting the Root Cause Correction Action Document will pay big dividends in customer happiness.

The Root Cause Correction Action Document’s components are fairly simple.   Here’s a brief description of each of the sections:

Heading

The heading section contains the customer name, document date, numeric document identifier, the date the problem occurred and the person preparing the document.

Problem Definition

A clear, concise definition of the exact problem.   You need to remember that not everyone reading your document will have a technical background.   Leave the technical mumbo-jumbo out of it.   You are trying to inform your customer of the event NOT confuse them.

Business Impact

What was the impact on their business?  Don’t sugar coat it.   Tell them that the failure caused a 14 hour outage on their production ERP system.   The business impact of the example shown below would be “A 37 minute delay occurred in replication between server ORAPGH and DB2DEL.  During this time DB2DEL reports did not provide current data.”

Event Timelines

This is a chronological timeline of the events that led to the problem (if you know what they are), and the steps that were taken to correct the issue.  Include every step that occurred up to and including verifying that the fixed system was indeed operational.   Here’s an example:

Wednesday May 5, 2010

18:08    Remote DBA Experts’ log monitor for the replication engine determined that replication was not successfully occurring between the two production platforms (ORAPGH and DB2DEL)

18:10     Remote DBA Experts notifies Delaware business units that data replication has stopped and reports being generated will not be current.

18:15     As a recommended action previously provided by the software vendor, Remote DBA Experts stopped and started the replication engine on both platforms.

18:30    Remote DBA Experts verified that the replication engine was running and restarts replication processes.

18:45     Remote DBA Experts verifies that replication was successfully occurring between ORAPGH and DB2DEL.

18:47     Remote DBA Experts begins monitoring the delay to determine the length of time it will take the replication engine to resynchronize the data between ORAPGH and DB2DEL.  Delay estimation is calculated to be 15 minutes.

19:00    Monitors show that both environments are synchronized.

19:05    Remote DBA Experts notifies Delaware business units that replication is occurring and all data is current.

19:10    Logs and trace files are collected and a Severity One problem is initiated with software vendor.

Thursday May 6, 2010

07:00   Software vendor contacts Remote DBA Experts support personnel.  States that problem was caused by a previously unidentified software bug.  Recommends upgrading product to newest release (we’ve never heard that one before).

Problem Root Cause

The underlying causal factor that created the problem event.  In the case above, the root cause was due to a software code issue that caused replication to terminate abnormally.

Contributing Factors

There are times when the problem is exacerbated by contributing factors.  In our example, if a long running job prevented us from successfully stopping the replication engine (leading to a longer outage), we would include a description of that issue in this section.

Resolution

This section contains the actual steps that were taken to correct the problem.   It does not restate the steps in chronological order.   It is a brief description of the activities taken to correct the issue.

Future Prevention

The Future Prevention Section is the most important component of the Root Cause Corrective Action document.  This section provides the steps that you will take as a service provider to ensure that the problem does not reoccur.    It contains a list of action items, the person responsible for completing that action item and a date the action item will be complete.

Signature Section

Signed by the technicians involved with the problem and a member of the service provider management team.

The Importance of Following Up
If you have been reading my previous blogs, you know that we feel so strongly about customer feedback at Remote DBA Experts that we have created a customer feedback strategy called “The Customer Feedback Engine.” We have established multiple communication flows to ensure that we receive feedback from all of the personnel that we support including management, DBAs, developers and end-users.

One of the key strategies is the role our Service Assurance Manager plays.  Remote DBA Experts’ Service Assurance Manager contacts all customers when a problem occurs.  Whether or not we caused the problem is immaterial.  We feel that is our responsibility as a service provider to let our customers know that their application’s performance and availability is important to us.

As my old boss Dan Pizzica used to tell me (when I was a VERY junior technician) “It really doesn’t make a difference who or what broke YOUR database (strong emphasis on the word YOUR). You are the technician who is ultimately responsible for fixing it. The buck stops with you. If you can’t protect your environments, you aren’t doing your job.” We all know he’s absolutely correct.

Wrap-up
It was not the intent of this blog to coerce readers into using the Root Cause Corrective Action information we provide to customers here at Remote DBA Experts verbatim.  It was to promote the benefits that a structured, well thought out problem notification document provides to all of us who are responsible for keeping our customer’s environments highly available and high performance.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

We also know that each organization, group and individual user has their own unique set of value drivers they we’ll use to evaluate the quality of service being provided to them.

It is the responsibility of the service provider to manage their customer’s expectations throughout the entire sales and service delivery life-cycle.   Customers that have a firm understanding of the service offerings, emergency response times, work request lead times and completion times have a much greater chance at being a satisfied customer than those that don’t.

Let’s face it, being a technical service provider isn’t an easy job to begin with.  Your customers will hold you to a higher standard than they do their own in-house personnel.    And they should.  Why complicate the customer management process by not providing your customers with a strong understanding of what they can expect from your organization?

Defining a set of clear, concise Service Level Agreements (SLAs) will help to solidify support requirements and dispel any incorrect assumptions a customer may have on the services your organization provides.  Your customers probably won’t be aware of your current workload and resulting work request lead times until you tell them.

I am a firm believer that every IT unit should meet with their internal and/or external customer base that they support to establish a set of measurable Service Level Agreements.  You don’t have to be a third-party services provider to take advantage of the benefits that a well-defined set of SLAs provides.   I have created a standard SLA document for each company I have been involved with, whether I was an internal employee, consultant or third-party service provider.

Our SLA document at Remote DBA Experts is fairly simple.   It contains Service Categories and individual services.   The SLA entry is clear and concise.   Each entry contains a description of the service, when we will provide the service (upon request, ongoing, reoccurring frequency) and the amount of time the customer can expect the service activity to be completed by.

Service Level Agreement Development:

• Service Identification – The first step is to create a list of services your organization provides to the internal and/or external customer base.   The Information Technology Infrastructure Library, or ITIL as it is most commonly known by, uses the term “Service Catalog” to describe the service listing. ITIL documentation states that IT organizations “must identify what internal and external customers expect” from the service organization before the Service Catalog can be created.   Identifying what your customers want is beyond the scope and intent of this article.   Our focus is to ensure that your customers fully understand the level of service your organization will provide to them.  It is important to list all of the services you offer.  Your SLA document may not contain a minimum service level for all of them, but each one must be evaluated.  You also need to identify if you are guaranteeing any minimum levels of availability and performance for products, applications, hardware components that your unit supports.
• Identify key Service Groupings for the SLA Document – Categorize your list of services into groupings of similar service activities.   At Remote DBA Experts, we have grouped our services into the following basic categories:
  • Customer Integration Activities – Our SLA contains a list of customer integration services that we will perform and their associated target dates.   Not only do we offer customer integration service levels to our customers, we have also defined a set of service levels that we ask all new customers to adhere to during the integration process.   Since we are a remote services provider, we need connectivity, accounts/passwords, environmental descriptions, etc.  Customers understand that we need this information before we can begin servicing their account.   Our SLAs state what we need from the customer and when we need it by.  The remaining service groupings are fairly obvious.
  • Monitoring – Robust monitoring is a key benefit that we provide to our customers here at Remote DBA Experts.  Our SLAs state what we monitor, how often we monitor a particular resource or event, and how quickly we will respond (15 minutes in our case) if a threshold is exceeded or a monitored event occurs.
  • Problem Resolution Activities - One of our SLAs is “Provide expert diagnostic analysis for all database operational problems that arise.”  The frequency is “Upon Problem Identification” and the Service Level is “Troubleshooting begins within 15 minutes of problem identification and continues until problem resolution.”  It tells our customers that when we identify a problem, we will be logged into their systems within 15 minutes and will work on it until it is fixed.   Pretty clear.
  • Database Maintenance and Tuning – This category contains a listing of the various service activities we will perform daily, weekly and monthly to ensure each database is highly available and has optimal performance.
  • Change Management – Our largest category contains all of the changes we can be asked to make to a customer’s database and/or schema.
  • Backup/Recovery and Disaster Recovery – We feel that this set of services is so important to our service offering that we have a service grouping that contains all of the key activities related to making sure our customer’s databases are available when they need them.
  • Installs/Upgrades/Clones – This category contains some of the most time consuming activities that we do for a customer.  The completion times are clearly stated.  We don’t want our customers to enter a ticket to clone their 14 tier Oracle E-Business Environment by the next morning.  Since we service multiple customers here at Remote DBA Experts, clearly stated notification and completion times for time-consuming projects are critical to our ability to service our entire customer base.
  • High Availability Configurations – RAC, Fail Safe, O/S, Data Guard and vendor provided high availability installation services are contained in this category.
  • Customer Management – This category contains all of the activities that relate to informing the customer of our activities.  Entries include meeting frequency, monthly reports, time reports, Root Cause Corrective Action discussions when problems occur and how quickly we will respond to a customer request for additional information.   If it is a communication activity or mechanism with the customer – its in there.

Helpful Hints for Service Level Agreements:

• Make every effort to create a complete list of service activities.  As stated previously, although every activity you identify may not be included in the Service Level Agreement, each one must be evaluated.
• Categorize your list of services into groupings of similar service activities.
• Don’t use high-tech verbiage to describe the service activity. Make the service description clear and concise.  Remember that your audience isn’t as technical as you are.
• During contract negotiations with external customers or meetings with your internal customers, take the time to describe each service entry.   Make sure your customers fully understand each line item.   If your customers find a particular service entry to be confusing or unclear, modify the entry and keep modifying it until it is clear.
• Routine reviews of your service entries and comparing them against changing customer needs are essential to ensure that your services and minimum service levels continue to satisfy customer requirements.
• Don’t create service level metrics for minimum thresholds (uptime and performance levels for example), that are outside of your control.    If you don’t completely control the environment, don’t guarantee a minimum threshold metric for it.   If you must, ensure that the verbiage of the service level metric description clearly states that your organization will only be held responsible for those activities that they do control.  For example, if you are a database unit, you don’t want to miss a minimum service level metric for application availability because of a network problem.  Guarantee only what you can control.

At Remote DBA Experts, we review all of our SLAs on a regular basis.  We also perform in-depth analysis on how well we are accomplishing each of our service level activities defined in our SLA documents.  We use these results to create an internal “report card” that we use to measure our success as a service provider.   If we find that we are weak in some areas, we take steps to correct the deficiency and quickly move on.

There is a direct correlation between how clearly a customer understands your service level guarantees and a successful customer/service provider relationship.  As I stated previously, you probably won’t be meeting all of your customer’s expectations if they (and you) don’t have a firm understanding of what they are.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

The general areas that are evaluated by the Service Delivery Alignment Engine include:

• General Support – How does the customer perceive the quality of our support?  What are we doing wrong?  But it is also important to identify what we are doing right.
• Responsiveness – Are we responding quickly enough to their requests?  Do we complete tasks when they are needed?  The number one complaint reported by numerous industry surveys, by an order of magnitude, is slow response to customer requests.   Think about the last time you had to wait for anything longer than what you thought was appropriate.   That is why service delivery organizations must create a robust set of Service Level Agreements to ensure customers are aware of the pre-defined turnaround times established at the beginning of the relationship.
• Communication - There are so many aspects to this subject, it is important to provide the participants with specific examples.
  • Frequency – Are we providing our customers with information on a timely basis?  Are we keeping them aware of our accomplishments with clear updates on large projects?  Do we keep them informed when we complete daily work requests in a timely manner?  Are we providing them with the appropriate status reports on long-running problems that are affecting their application?
  • Content - Does the level of communication we are providing to the customer match their technical background?  Is the language too technical, not technical enough, too high level or are we delving too deep into the details?
  • Clarity - When we are providing or asking for information, do they understand what we are telling them or what we want from them?
  • Communication Mechanisms- What communication transfer methods does the customer feel comfortable with?  Do they prefer e-mails, ticket updates, phone calls or quick face-to-face meeting?
  • Effectiveness - Are we performing the right tasks they need when they need them?  Just as you may find that you need to provide your customers with additional activities to support their needs, you may also be providing them with service activities that are not important to them.
  • Current Issues - Do they have any current issues that need to be addressed?

Realignment of Services to Meet Changing Customer Needs
Action plans to improve in weak areas are created during the evaluation process. Once agreed upon, the internal procedural and process changes can be implemented to tune and tweak the service delivery mechanisms.  The changes can range the spectrum, from increasing the volume of communications with the customer to providing additional services that were not previously required and thus not defined during the initial contract negotiations.  We understand that our customers’ needs change during the course of our relationship life-cycle.  It is up to us as the service delivery provider to quickly identify those changes and adjust our services accordingly.

If the realignment required is substantive, a formal document that describes the issues identified and provides details on the changes that will take place to address them is distributed to the customer involved in the evaluation.  Follow-up meetings, designed to ensure that the service delivery changes are having the desired outcome, are scheduled a few weeks after the action items document is distributed.

Wrapup
We continuously approach our customers to gather feedback to improve our services.   We also look for new, innovative ways to gather customer feedback, process it, improve our approach to service delivery and then communicate those improvements to our customers.

In my next blog, we’ll cover some of the mechanisms we use to collect customer feedback data.  We’ll begin with Service Level Agreements.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Since I work for a remote services provider, collecting, analyzing and processing customer feedback is critical to the success of our organization.  Our customers come in all shapes and sizes, from mom and pop shops to multi-national Fortune 100 Corporations.    Virtually every market vertical is represented – from retail to medicine and from heavy industry to high-tech.   We have learned that each one of our customers has a unique set of value drivers that they use to evaluate our services.    If we don’t understand what they are, how can we be sure that we are meeting their needs?

The answer is that we can’t.  And if that occurs, we have lost the battle to achieve our goal of 100% customer satisfaction.   We are in a very competitive business line.    Our customers’ needs change constantly.  If we don’t execute, one of our competitors will be more than happy to step in.

But do support personnel that work for a single business organization need their own “Service Delivery Alignment Engine”?   The answer is ABSOLUTELY.  The engine can work for any unit that delivers a service that needs to be constantly tuned, tweaked and adjusted to meet their customer’s changing business needs.

Over the next two posts, we’ll break the RDBAE Service Delivery Alignment Engine into its basic core components to better understand how they work together to accomplish the goal of 100% customer satisfaction.

Feedback Mechanisms:
We have learned that multiple communication paths are required to feed the Service Delivery Alignment Engine.  Relying upon a single path does not provide us with the volume and diversity of feedback we need to meet our customers’ unique set of value evaluation drivers.  Since many of these mechanisms are fairly complex in their implementation, they will warrant separate blogs to discuss them.   We’ll review them at a high-level in this blog and dig deeper into some of the more complex individual mechanisms in upcoming blogs:

• Service Level Agreements – Effective measurements are required to evaluate the success of any activity.  But what do you measure the results against?   What is the baseline?  The baseline for any service delivery organization begins with a clear set of Service Level Agreements.   The agreements are a set of predefined service performance objectives that are mutually agreed upon during the initial stages of the relationship life-cycle.  They describe and define a set of measurable targets that are used to evaluate service performance.   We’ll discuss some of the individual core service delivery measurement components in my next blog.
• Ticketing System – Ticketing systems are used by customers to initiate work requests and track progress.  The ticketing system also contains several key pieces of information that can be used to track customer perception of responsiveness and work quality.  Are the work requests being completed by the requested completion date?  Is the customer following the predefined set of service level agreements that were agreed upon?  Does the customer have to continue to supply additional information needed to process their requests?  Is the technician who is processing the request providing the appropriate level of feedback when a milestone is achieved or the request is completed?
• Daily Feedback – Remote Database Experts, like most service delivery organizations performs dozens (and dozens) of administrative activities daily.   Each of those individual activities is evaluated by customer personnel on a daily basis.  These individual evaluations are then combined together and used by the customer to evaluate overall service performance. A constant flow of information from personnel responsible for those daily changes is provided to the team responsible for overall customer satisfaction.   Issues with support quality are addressed immediately.  Particular attention is paid to long running problems that are affecting application performance or availability.
• Weekly Customer Meetings – Weekly meetings are held by personnel who are servicing the account on a daily basis.  The discussion focuses on current and upcoming activities.  We always ask three questions at the end of every discussion.  Is there anything else we can be doing for you? Is there any service or activity that we need to improve upon? What are we not doing that we should be doing?
• Service Level Assurance Discussions – The customer management team that is responsible for the relationship is also interviewed on a regular basis.  We understand that the management team’s expectations may differ from those held by customer personnel that are involved with our organization on a day-to-day service delivery basis.
• Root Cause Corrective Action Reports (RCCARs) – A customer that is affected by an application outage or slowdown needs to have a firm understating on  what caused the problem, the activities performed to correct the problem and the action items that will be undertaken to mitigate or prevent the problem from occurring again.  The Root Cause Corrective Action Document provides information on the underlying causal factors that generated the problem and a timeline of events that occurred during the problem event. This ensures that all problems are properly analyzed and that all steps are taken to prevent future occurrences.  This is a key component of our problem resolution strategy in addition to obtaining customer feedback on the quality of our problem resolution capabilities.    I’ll dedicate an entire blog to the Root Cause Corrective Action Report Process.

Next week, we’ll discuss the remaining components of the Service Delivery Alignment Engine: Processing customer feedback and realigning services to meet changing customer needs.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Verbal and Written Communication Skills
I think people read this blog because they take pride in their work and want to become better at their chosen profession. So here’s my second piece of non-technical advice: The importance of improving your communication skills can not be understated. I don’t care how strong of a technician you are, if you can’t communicate effectively with your peers, you won’t be able to succeed in this profession. In the old days, you might have been able to get by with just your technical skills. That is definitely not the case in today’s business world.

Take a look at your last performance appraisal forms, I’m betting that most of the criteria you are being judged upon depends upon communications. The key words and phrases to look for are “ability to work in a team environment”, “keep supervisors informed”, “maintain good communication with the user community”, “ensure the content of the communication is at the appropriate level for the intended audience”, “provide system and user documentation for projects and system enhancements.” I pulled all of the aforementioned phrases verbatim from one of my own past performance appraisals. I reviewed all of the criteria that I was being evaluated upon and found that almost ninety percent of the items depended upon verbal or written communications.

If you don’t have good communication skills, all is not lost. Like anything else, these skills can be learned. I still consider myself to be only a fair writer. I am in awe of people like Craig Mullins who can just sit down and let the words flow. I often find myself agonizing over every word and sentence. When I first started working in a corporate environment (20 years ago now), my writing skills were terrible. My original career was not database administration, it was construction—a job that didn’t require you to excel at written and verbal communications. One on the job accident, 9 operations and 11 months of vocational rehabilitation training later and I had a new career as a COBOL programmer. I went from working with a construction crew to working with computer programmers. My first employer was a very large and somewhat stuffy financial institution. When I was employed there, men couldn’t leave their floor without wearing their suit coat.   That’s the way it was in the 80′s.

I quickly learned that getting caught wiping your computer screen off with your tie didn’t really show your managers that you were good at thinking “outside of the box.” That was about the only thing I thought that piece of knotted cloth around my neck was good for. For the first six months, I refused to tie them. Being the non-conformist that I was, I just loosened them up, slipped them off and hung them up. Saying that my communication skills were rough around the edges when I started my career would be an understatement. But I had the good fortune of having a manager that understood the importance of both verbal and written communications. I would write a memo, she would correct it with her red pen and send it back to me for a rewrite. Many of them had a “Nice Try!” and a smiley face on top.

After becoming exasperated because of the numerous rewrites (and seeing all of those smiley faces), I thought I had better improve my writing skills. I read books, practiced writing, and became involved with as many company newsletters and related communications as I could. When I asked to join a newsletter, I always started with “I’m not the greatest writer, but I’m trying to learn.” I also asked my peers that worked on the newsletter to critique my work. The more I was critiqued, the better I became.

The same was true with public speaking. My first speech could be described as being “somewhat less than stellar.” Craig Mullins would gently prod me from time to time until he finally convinced me that speaking was something I should be doing from time to time. Craig promised to sit in the back of the room for my first speech and give me hand signals if I was speaking too fast, too slow, too loud or too soft. Halfway through the speech, my knees were knocking and his hands were in constant motion.

I found that like anything else, experience helps. But I will say that my speaking career was not without excitement. I learned that you really shouldn’t drink a carbonated beverage wearing a tie mike that is attached to a set of 6 12-foot speakers. When I was done chugging the pop before the speech, I looked around and saw everyone laughing at the noises I had just made.

I also learned that some podiums are on wheels and those wheels aren’t always locked. I started my first sentence, leaned against the podium, and it began to move. I tripped a little trying to stop the podium from moving and ended up heading for the end of the stage at a very rapid rate. It was a raised stage too, about six feet higher than the first row of seats. As I quickly approached the end of that raised stage, I noticed that the people in the first row were making motions just like the extras did in the old Godzilla movies- right before they got stomped on. Lucky for me one of my work buddies in the first row had the good sense to jump to his feet and stop the podium (and me) from killing a few members of my audience.

At a very large conference, the speaker (who now works for a competitor of mine), that was using the room before my session, left with the tie mike.   The moderators and technicians searched but couldn’t find a spare in time.  What they did find was a 4 foot corded mike that they plugged into the middle of the floor.   I then gave half my speech to 500 participants bent over at the waist.  Luckily, they rounded up the cordless version.   I got over those little snafus and kept plugging away. With each subsequent speech, I started to improve.

The point I am trying to make is that you can improve upon your communication skills. IT shops are no longer evaluating technicians purely on their technical skills. I have seen the soft skill evaluation pendulum swing a little more each year. It is the total package of skills that you bring to the table that you are being evaluated upon. We all know the importance that our technical skill sets have upon our success in this field. But you also need to be well rounded in all of the skill sets your managers are looking for.

In my next set of blogs, we’ll look at some of the different mechanisms we will use to communicate to our customers.  Whether your customer is across the hallway or across the globe, these mechanisms can be used to coordinate your activities with others and keep your customer’s informed of your progress.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

I use change management as a simple example.  Oracle states that 90% of all database failures can be attributed to human error.   Poor communication is certainly one of the problems that we can categorize as a human error.    The last thing you want to hear from an operator when you are requesting the restoration of a tape backup to a different disk is, “I thought you wanted me to restore the files in their original location – not to that spare disk.   I think I just overlaid your prod database…”

Remote DBA Experts provides remote database administration services to organizations across the globe.   We perform 100% of our administrative activities to customers that we rarely see in person.   We have become experts at effective verbal and written communications.   So, let’s talk about communication skills and the important role they play in our careers.

This two-part blog entry will be a somewhat lighthearted look at my own life’s lessons on effective communications (or lack thereof).   In future posts, we’ll look at different mechanisms we can use to communicate and coordinate more effectively with others.  We’ll also learn how we can use effective communications to keep our customer base happy.

It used to be that you could get by with a total lack of interpersonal skills in our profession. Well those days are long gone. If you want to succeed, you’ll need to learn how to communicate effectively and play well with others. These posts contain a few pearls of wisdom, and a story or two, to help you become a well-rounded technician that is on the fast track to a successful career.

The DBA’s Evolving Role
You have read numerous articles on the changing role of the DBA. How the various database vendors are making their products so easy to administer that, sooner or later, DBAs will no longer be required to be expert technicians (or be required at all). If you have been reading this blog, you’ll know that I totally disagree. It is my opinion that database administrators will always need to be just as technically proficient as they had to be in the past. I will agree that database environments are becoming easier to administer. Oracle Grid Control allows us to administer, monitor, tune and troubleshoot an Oracle database without having to go as deep technically as we had to in the past. Although, we won’t have to know as much about the internals as we used to, our skill sets will become much broader in scope.

The database engine is taking on a much more strategic role in most organizations. It provides the mechanism to store physical data along with business rules and executable business logic. The entire application environment (data storage, business rule enforcement, application program storage, communication, system monitoring) is now controlled by the database. Over time, the database engine will store more information related to the understanding of the business, the meaning of the data stored (metadata), and the mechanisms to control and track versions of the database, access programs and related software. As the database’s area of influence expands, so does the DBA’s. Can any one of us predict what IBM, Oracle and Microsoft have up their sleeves in their next “latest and greatest” release? Not me. But THAT is what makes this job exciting. Our area of technical influence will be expanding, not contracting.

The Importance of Soft Skills
Now that I have expressed my opinion on the expanding role of the DBA, let’s talk about the importance of soft skills. Before we begin, I define soft skills as the ability to communicate in both written and verbal forms and the ability to interact with fellow employees in a positive manner. In the past, a technician’s lack of interpersonal skills was often overlooked. The more technically proficient the technician was, the more leeway he or she was given. Let me provide you with a couple of quick examples.

When I first started in this profession, I attended a meeting at a large financial institution that included some pretty high-level representatives from both the business and technical areas. One of the technicians that attended the meeting was a mainframe operating system support technician. The guy was well known to be very good at his job and as nasty a person as you would want to meet. The meeting started and it quickly became apparent that most of the issues being discussed would be about business processes. The O/S expert slammed his pen and pad down on the table and declared, “It looks like this is going to be a waste of my time. I have work to do. Call another meeting with me when you need technical advice.” He then promptly walked out. Being a junior level programmer, I was in awe. I asked my boss after the meeting who he was. He stated “one of our mainframe gurus, don’t act like he does until you get as good as he is.”

A dozen years later, I saw a technician raise his hand at an enterprise-wide IT meeting when the CIO asked (rhetorically I think), who was the most important person in the organization. I think the CIO thought the answer would be “the CEO.” The techie who raised his hand said “I am.” I think every manager attending that meeting shrunk down in his or her seat. When the CIO asked him why, he stated, “When my computers go down, all business stops.” At a previous job, I had a 20-minute conversation with a UNIX admin who never bothered to turn around from her screen to look at me.

Although my examples may be over the top, they show you the mindset that often plagues our profession. As the years have gone by, I have migrated from DBA to DBA Unit Manager and now DBA Operations Manager. I have seen too many excellent technicians end up with a mediocre career because they achieved a reputation for “not playing well with others.” I talk from experience. If you want to excel as a technician, you will need to be technical, but you will also need to work well with others.

Next week, in the conclusion of this two-part blog entry, we’ll continue our discussion on the importance of communication in achieving customer happiness with more words of wisdom and personal experiences.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Since we are a remote services provider, our customers’ businesses range the spectrum from heavy industry to high technology.   We understand that each of our customers has a unique set of value drivers upon which they evaluate the quality of services we provide to them.  We are also aware that our customer’s service delivery expectations are constantly affected by changing business and economic drivers.  The types of services we provide must be tuned and tweaked continuously to ensure that we meet these fluid service delivery expectations.

Once you have collected your own set of customer evaluations, you need to review the results and tally them to identify any common issues that are occurring. Epi Torres, my Remote DBA Experts’ co-blogger, has already identified the key evaluation criteria that are used to evaluate services providers: Timeliness, Efficiency, Effectiveness, Responsiveness, Quality and Integrity. You can categorize your own customer responses according to the key evaluation criteria we have identified or create your own. Once you create the categories, your next step is to perform a thorough analysis of the responses in each of the them.  If you identified a common theme of recommendations, your next step is to rank them in order of importance to your customers.

As you evaluate the categorized responses, you will need to take external influences into consideration that may be affecting your ability to provide high-quality support (inadequate staffing levels, time consuming projects, technical limitations of the environment). One of the key benefits of these documented responses is that this information can be passed up the management chain for further analysis and recommendations. For example, if you have too few DBAs and too many work requests, this additional documentation will notify your management team that the quality of work is suffering.  If additions to staff are hard to come by due to economic conditions, you have at least made both your management team and customer base the importance of setting realistic deadlines and prioritizing workloads.

Action plans to improve in weak areas are created during the evaluation process. Once agreed upon, the internal procedural and process changes can be implemented to tune and tweak the service delivery mechanisms.  A formal document that describes the issues identified and provides details on the changes that will take place to address them can then be distributed to the customers involved in the evaluation and the DBA team’s management chain.

It is important to also address specific customer recommendations that were contained in the customer responses.  Even though a specific issue was not identified as a common theme that exists across the entire customer base, they must still be addressed to ensure that individual customer’s happiness.  Is the customer’s expectation realistic within the confines of the external influences?  If it is, the recommendation needs to be addressed.

Follow-up meetings, designed to ensure that the service delivery changes are having the desired outcome, should be scheduled a few weeks after the action items document is distributed. The meetings should be held with the customers that participated in the original review process.  The discussion will address recommendations that were identified to be common amongst the entire customer base as well as the individual recommendations identified by that particular customer.

The process then becomes iterative in its nature. We have developed our own cyclical customer feedback process here at Remote DBA Experts.  We feel so strongly about this iterative approach to customer feedback that we have created a standardized customer feedback strategy called “The Customer Feedback Engine.” We have established multiple communication flows to ensure that we receive feedback from all of the folks that we support including management, DBAs, developers and end-users. We continuously approach our customers to gather feedback to improve our services.   We also look for new, innovative ways to gather customer feedback, process it, improve our approach to service delivery and then communicate those improvements to our customers.

You need to know what your customers are expecting from you.  You also need to know how they feel about the quality of support that you are currently providing. Without that information you will never be sure if you are providing the highest level of quality support possible.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Since we support many different organizations in virtually every market silo (high technology, heavy manufacturing, retail, health care, etc.), it is obvious to us that each of our customers has a unique set of value drivers that they use to evaluate the quality of our service.   If we don’t understand what they are, how can we be sure that we are meeting their needs? The answer is that we can’t.

Since I work for a remote services provider, we live and die by customer feedback.   But do DBAs that work for a single business organization need their own “Customer Feedback Engine”?   The answer is ABSOLUTELY.

Each group that you support has their own set of value drivers.  You have to understand what they want.  If you don’t, you will be viewed as just a mere technician and that is exactly where you’ll stay throughout your career.  This is NOT an optimal career path. Database administrators have a highly visible role in every organization. You can take advantage of that role to be viewed as someone who is a key player, as opposed to a mere technician.

You can obtain customer feedback by any number of physical mechanisms but it all boils down to this.  You need to put your ego aside, understand that you need to obtain constructive criticism to get better at what you get paid to do and ask your customers. Some of the more popular choices are DBA “report cards” or surveys that are sent to your customer base.   I have used report cards with much success in the past.

A helpful hint with report cards is to make sure you include a field for the participant to include their business unit but not require a signature.  This will help the participants feel more comfortable when they fill out the survey and provide you with a higher quality source of feedback information that you can use to tune and tweak your service delivery methods.

Since we support so many different customers, let me give you some examples of the questions that you need to ask:

• General Support - The intent of the first question is to obtain a general, high-level understanding of how you are doing.  It opens up the conversation to allow you to obtain feedback that may not be obtained from the more specific questions you will ask.
• Responsiveness - Are you responding quickly enough to their requests?  Do you complete tasks when they are needed?
• Communication - There are so many aspects to this subject, it is important to provide the participants with specific examples.
  • Frequency - Do you provide your customers with information on a timely basis?  Are you keeping them aware of your accomplishments with clear updates on large projects?  Do you inform them when you complete daily work requests in a timely manner?  Are you providing your customers with the appropriate status reports on long-running problems that are affecting their application?
  • Content - Does the level of communication you are providing to your customer match their technical background?  Does the language you use seem too technical, not technical enough, too high level or are you delving too deep into the details?
  • Clarity - When you are providing or asking for information, do they understand what you are telling them or what you want from them?
  • Communication Mechanisms- What communication transfer methods do they feel comfortable with?  Do they prefer e-mails, ticket updates, phone calls or quick face-to-face meeting?   Our customers range the spectrum.  From those that blatantly tell us “never call me, just e-mail and don’t do that a lot” to customers who prefer a continuous level of communications using all of the communication methods available.
• Effectiveness - Are you performing the right tasks they need when they need them?  Just as you may find that you need to provide your customers with additional activities to support their needs, you may also be providing them with service activities that are not important to them.
• Quality of Support - Ask them to rate the quality of the services you are providing to them.
• Current Issues - Do they have any current issues that need to be addressed?
• Additional Information - What other questions should you be asking them?

As we learned in this blog, effective measurements are required to judge the success of any activity.  The quality of support you provide needs to be reviewed on a regular basis.  These questions allow your customers to provide you with important feedback on the quality of your support.  You can then “tune and tweak” your services accordingly.

Meetings should also be held with the customer groups that were asked to participate in the analysis.  It is important to make your customers aware of the external influences that may be affecting your support (inadequate staffing levels, time consuming projects, technical limitations of the environment).  These aren’t intended to be excuses but your customers may not know that some of their expectations will be hard to achieve until you provide them with the reasons.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Whether your customers are internal or external (like mine here at Remote DBA Experts), if you focus on just providing great technical support, you are only winning half the battle when it comes to keeping your customers happy.

One of the benefits that I am able to provide is that my organization supports dozens (and dozens……) of remote database administration services customers.   Our customer base spans the market spectrum from high technology to heavy industry.  In addition to having every market sector represented, the size of our customer base also varies widely.   We support organizations that have virtually no IT staff members to multi-national organizations consisting of huge IT staffs and hundreds of database servers.

This gives me the unique perspective of having a broad knowledge of what database support services customers want.   Name a market sector and I would be highly surprised if we didn’t have at least a handful of customers representing it.  Our customers’ technological strategies also vary widely.  Many of our customers don’t want to push the technological envelope while others want to stay at the forefront of every new database feature.

As I stated earlier in this blog, you need to be more than just a technician to be a success.   If you focus just on being a good technician, that is how you will be viewed and treated.   Being a mere technician or “table jockey”, as we call low strategic value DBAs at RDBAE, will not lead to a successful career path as a database administrator.   I’m responsible for service delivery at Remote DBA Experts.  When a customer views my organization as nothing but a pack of “table jockeys”, I have made a big mistake and have jeopardized that relationship.  If a customer sees us as a “utility provider” (no added value, just keeping the lights on), I’m in trouble.   In order to ensure our customer’s happiness, being a “table jockey” or a “utility provider” isn’t enough.  Providing excellent technical support is the foundation of a good relationship and building credibility, but it is only one facet.

Is this just a “remote DBA thing”?  Absolutely not!   You know that your customers expect you to be the technical expert.   If you aren’t at the top of your game technically, you will never obtain credibility with them.   But after you obtain that technical credibility – what next?   That will be the focus of my next set of blogs.  Will this require you to change your entire strategy? Once again, absolutely not.  Here’s a great quote that I often refer back to:

“Small changes can produce big results – but the areas of highest leverage are often the least obvious.”

- Peter Senge, The Fifth Discipline

But where do you start?   How do you determine how you are viewed?  Well, the first thing is you need to know where you are starting from!

It is obvious that effective measurements are required to judge the success of any activity.  In my next blog, we’ll discuss a DBA report card.  Questions that I have been using for years.  We’ll review some of the information you will need to gather in order to determine where you are when it comes to being a strategic-value DBA.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

We should all know by now that the preferred administration and tuning tool of choice is either GRID or its standalone counterpart, the Database Console.   The advanced alerting, monitoring and administration features make these tools invaluable to Oracle DBAs. The advanced performance monitoring and analysis features should also make them your number one tool for performance problem determination.

The future of Oracle database tuning will be administrators interpreting and implementing the recommendations generated by the intelligent advisors and ADDM. It is a foregone conclusion that Oracle will continue to improve upon their performance monitoring and analysis toolsets. Self tuning features are no longer options that are “nice to have”, they are requirements for Oracle’s competitive survival. Microsoft SQL Server is continuing to scale, moving into areas that were once dominated by UNIX big-iron machines running Oracle databases. Oracle must compete with SQL Server’s ease of use or it will undoubtedly lose market share.

As the intelligence of the advisors and ADDM increases, the need to possess an in-depth knowledge of Oracle and the requirement to review reams of detailed diagnostics to improve database performance will decrease. And you heard it here first folks, I also think that reading SQL traces and statistics dumps will be a thing of the past. My crystal ball tells me that its just a matter of time until Oracle’s SQL advisors make SQL traces and statistics dumps less and less important until they become totally unnecessary.   Tuning should NOT be a high ROI activity, it should be delegated to a mere maintenance activity.   There is just too many other things for us to do.  IT shops will continue to demand that DBAs be more than just technicians.   Our role will be how to utilize the ever-increasing number of features provided by the database to solve business problems.   NOT looking at a 47 step access path.

The new breed of top tuners will be the administrators who focus on how to use the toolsets and interpret their output. Not the tuners who spend the majority of time digging down into the dark, inner workings of the Oracle software. I’m not saying that knowing how the database works is immaterial. I am stating that this intimate knowledge will become less and less important as the tools mature. It’s only a matter of time until tuning is relegated to a minor sub-task that can be scheduled between other activities.

There will always be a breed of “super tuners” that write articles, books and speak at various events.   But this segment of our population will be forced to focus less and less on the core internals of the database. The scope of the information they disseminate will need to broaden to cover new tuning toolsets and performance enhancements in the “latest and greatest” release of the Oracle product set.

If we look at a history of Oracle database enhancements, we can easily see that the vendor is identifying facets of the database that are complex or problematic and providing solutions for them.   Remember manual rollback segments?    How big were they supposed to be?  How many?   Oracle recognized that DBAs didn’t do that great of a job administering them and decided that the database was better off doing that job on its own.   How about freelists? Dictionary managed extents?  Multiple settings for sizing individual SGA components?  Log checkpoint tuning?    There is no doubt that managing the core Oracle internals is becoming easier.

I also think that Oracle will eventually become self-tuning.  Personally, it can’t come soon enough for me. Don’t get me wrong, I enjoy tuning. I’m actually a fairly accomplished tuner.

But as the director of a remote services provider that manages thousands of Oracle, SQL Server, DB2 and MYSQL databases, the less time our product support teams spend on tuning, the better. If we do tune, our focus is on proactive and not reactive tuning.   The remote database administration field is a tough job.  Our customers hold us to an extremely high standard.   I’m OK with that.  They have turned over the keys to their most valued corporate assets – their data.  As database administrators, we understand the gravity of that decision and we don’t take the responsibility lightly.   The less time we spend on tuning, the more time we are able to spend on being proactive, using technology to solve business problems, adding strategic value, improving communication and,  in general, just building a positive rapport with our customers.

And just because we won’t be spending time performing proactive or reactive tuning, we all won’t be out of jobs. We’ll just be doing different things. I have listened to various industry pundits proclaim that the next release of so-and-so database was going to no longer require DBAs for support. Bull. I knew it was bull then, it is bull now and it will be bull in the future. Database companies know that they must add new features to be competitive. Every new release contains so many new features that I feel like I have to learn to support the database all over again (which is why I like this job, by the way).

Oracle now allows us to administer disk, lash multiple computers together to enable grid computing and the list goes on, and on, and on…. It took me four different articles to cover just a subset of the new features available in Oracle’s latest release. The database may become easier to administer in some areas, but there are so many new features being incorporated into the product that administrative duties are just migrated to other areas. Oracle ASM allows us to administer our own disk. How many Oracle DBAs could have predicted that they would have the capability to administer their own disk storage subsystems? Can anyone of us predict what IBM, Oracle and Microsoft have up their sleeves in their next “latest and greatest” release?  Not me. But THAT is what makes this job exciting.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery