The Art of Being a Successful DBA – Good Documentation is Essential

Let’s start our series on Art of Being a Successful DBA by covering the art of good documentation.   Although the importance of a well thought out and detailed documentation library is blatanly obvious, creating documentation is the task most often postponed by an overworked DBA unit.

Documenting processes, procedures and best practices is a task that is often considered to be boring and mundane. Most DBAs would rather perform virtually any other activity than sit in front of a screen using a word processor. As a result, creating documentation is often postponed until the DBA has a little free time to kill. Today’s database administration units are operating with smaller staffs, tighter budgets and ever-increasing workloads. The end result is that the documentation is either never created or created and not kept current.

But a robust detailed documentation library creates an environment that is less complex, less error-prone, reduces the amount of time DBAs spend learning new database environments and reduces the overall time spent on day-to-day support activities. DBAs are able to spend more time administering the environment than finding the objects they are trying to support and the processes and programs used to administer them.

The nature of my business as a remote services provider demands excellent documentation. The majority of environments we administer weren’t designed by my organization. The only way that we can ensure high quality and high-speed administration of these environments is to document them thoroughly. We document everything, from initial connectivity and customer contact sheets to detailed information on batch job streams and individual program execution. If we need to be aware of it, we have it documented.   We have dozens (and dozens) of internal best practices also documented.   If you are evaluating remote database services providers, you need to ask them for examples.

Documentation is also the foundation of many of the other disciplines I will be discussing in future blogs. Let’s continue our discussion with a few helpful hints to get you started.

Understanding the Important Role Good Documentation Plays
We all generally understand the benefits that documentation provides. I think that all readers will see the importance I personally place on documentation in upcoming blogs. Virtually every topic that I will discuss, from repeatable processes to paranoid administration best practices, I will be mentioning what to document and the important role that documentation plays.

Let me reaffirm my opinion in this one sentence:

Documentation is one of the foundations for virtually everything I will be discussing in this series on The Non-Technical Art of Being a Successful DBA. Enough said.

Creating an Organizational Environment That Fosters Good Documentation
I have been leading database teams off and on for 15 years now.  It is my responsibility as manager to create an environment that fosters the production of robust and high quality documentation. Let me describe some of the challenges that I have faced in the past and how I have overcome them.

Since I view high quality documentation to be my responsibility as the director of database services here at Remote DBA Experts, I ensure that it becomes part and parcel of every DBA’s job description.    If we feel that a candidate DBA doesn’t value the importance of documentation, we don’t hire them.    It’s OK if they don’t like to do it, they just need to express that it is something that is required to create a high-quality support environment.

I will add time for documentation when I estimate the amount of time it will take me to perform an administrative task during project planning meetings. I don’t settle for “we can do that after the project is complete” as an answer from an application project manager.

If I get overruled and lose one battle, I still make sure I document my activities after the project’s critical path is complete. And I don’t let a single setback prevent me from telling everyone in the next planning meeting that I need so many hours to document what I have done.

If you continuously sell the importance of documentation sooner or later you will begin to wear your opponent’s down. Although I prefer to call it “being relentless”, I’m sure that many of the application development managers (and my own managers) viewed it as “being a ….” (insert your favorite description here).

Every document I have created that provides a list of activities I , or my unit, needs to perform during a project has documentation included. It helps to integrate it into the fabric and culture of my organization’s environment.

Making Documentation Second Nature
You also need to ensure that generating documentation becomes a natural part of your daily activities. You must continuously remind yourself that documentation is a primary and integral part of providing high quality support services to your customers.

You must also remind yourself that it makes your job easier and benefits your fellow DBAs. You know what has really irritated me the most during my career? When a fellow DBA needs to be out of the office for a time and asks me to “help them out” by performing a complex, application specific administrative activity and then tries to verbally tell me how to perform the 300 steps it takes to do it.

Ever try to refresh an ERP application test environment from production when that test environment doesn’t have enough space to hold all of production’s data? 4,000 steps later and you begin to second-guess your choice of professions. That was the exact request from one of my fellow DBAs a few years ago when I worked for a large aluminum producer here in Pittsburgh. Not only did he get me to do the refresh but I also had to document the process for him along the way. Some call that being a good coworker; I would view that as having a big sucker taped to my forehead. Live and learn, I guess.

The moral of this story is: If you don’t want to be the only one that can perform that 900 step ERP Application production to test refresh, document it! If you don’t want to be called by the on-call DBA because he doesn’t know exactly where to add a datafile in an emergency situation (like an application developer forgetting to tell youu that they were loading 10 million additional rows into that 100 row table), document it! The more you document, the easier your life as a DBA becomes.

I’ve never had a great memory. It makes documentation easy for me. I also like to write and that helps. But I will admit that there are times that I would rather perform virtually any other activity than document.

But it has become easier because I continuously reaffirm to myself the importance of documentation. The more you reinforce that to yourself, the more second nature (and easier) it becomes.

Making Documentation Easy
I’m a huge fan of documentation templates. Here at Remote DBA Experts we have, or are in the process of creating, templates for everything we document. We have templates for documenting connections to our customer’s environments, their backup and recovery environments, their application specific processes to name a few. If it needs to be documented on a regular basis, we have a template for it. We also have generic templates for documenting environments and activities that don’t fit into other templates.

Microsoft Word document templates provide many features that streamline the documentation process and help to improve the quality of the content they store. I try to take advantage of as many features as I can. I use drop down selection menus, check boxes and radio push buttons to improve the speed and quality of the documentation process. I also take advantage of the help pop-up feature that Microsoft Word provides to create a detailed description of what information is to be entered into that field, check box or radio button.

I created my first template close to 15 years ago. It wasn’t created to document a process, but to document a change request. I’ll be covering change management in an upcoming series in this blog. Was the document crude? You bet. But it did the job. It has gone through numerous iterations and migrations from one Word release to the next. I continue to use it at organizations that don’t have a formalized and documented change management process. And you can’t believe how many I have come across that don’t.

There are dozens of software companies that offer content management solutions. Database vendors have also recognized this as a lucrative market. All of the major database vendors now offer advanced content management software, each one trying to outdo the other in the number of bells and whistles that their product’s offer.

Do a quick search on Google for documentation content management software and you will find out just how many competing products there are.

Content management products offer check-in/check-out features, document versioning, web portal access and advanced workflow capabilities to name just a few of the features desgined to improve content management. The competition in the content management market space is fierce to say the least. Content management vendors know that continuously adding new bells and whistles to their products is not just important for increasing market share; it is critical for their survival. Product costs can range from thousands to tens of thousands of dollars (or more).

If you have the funds and your management understands the benefits that a full-blown content management package provides, by all means begin a content management product analysis. But if you don’t have the funds, create a shared drive on your network and declare it to be the “DBA Documentation Portal.” We will be using a combination of Microsoft Sharepoint and Sales Force as our own DBA Documentation Portal.  Sales Force’s interaction with our document libraries provides us with a highly secured, highly customizable document storage environment.  It is our “one stop shop” for customer information.

What to Document
By all means this is not an all-inclusive list of what can be documented. Consider it as a starter kit to help you begin your quest for “documentis nirvanas”. Is some of this overkill for your particular environment? Maybe, but just consider this a general, high-level list.

Database Environment Documentation

• Naming conventions
• Servers (server names, operating system release, hardware vendor)
• Databases (vendor, database version, features enabled)
• Directories (everything from Oracle software installations that don’t follow Oracle standards, application code storage to personal script locations). If you, or a fellow DBA needs to find it, document where it is!

Application Specific Documentation

• Application type (i.e. data warehouse, online transaction processing, decision support, third-party application name and functionality it provides).
• Business unit requirements and related information for supported databases
• Uptime requirements (i.e. 24 X 7, 8 X 5)
• Database downtime windows
• Critical job processes
• Business unit and application developer contact lists
• Turnover windows for database changes
• Problem notification and escalation procedures
• Security Sensitivity. How sensitive is the data?

Process Documentation

• Repeatable administrative processes (covered in an upcoming blog)
• Backups – Probably the most critical set of documentation you will ever create. Document how it is backed up, what scripts back it up, where the back up is going to, tape retentions and backup message directories. If it is involved with a backup, DOCUMENT IT. Review the document with other units that are involved in the backup and recovery process. It is your responsibility to ensure that you don’t hear an operator say “What retention period? Nobody told me we were to have a retention on these tapes.” when you are in a recovery situation. Remember that Oracle states that human error, including miscommunications, is responsible for over 90% of failed recoveries. If you want to reduce recovery failures DOCUMENT THE PROCESS AND REVIEW IT.
• Oracle utilities. Document Oracle Import/Export input and output directories and where the files are that execute the utilities. The same with Oracle’s load utility. Document the execution scripts and input and output directories. A short note on why you are running it is also required
• Anything else you run on a regular basis to support a specific application
• Change Management. I’ll be spending an entire blog, or two, on this
• A daily monitoring activity checklist to ensure that no activity is missed. We have daily, weekly and monthly activities that are to be performed for each of our customers
• Assuming on-call responsibilities.  You need to make sure you are prepared for the on-call process.   The checklist should include what other DBAs are working on during your on call rotation, customer outages taking place, connecting to environments that are hard to connect to, etc..
• Complex administrative activities performed regularly
• Test and reporting database refreshes
• Data reorganizations
• Disaster recovery tests. The processes required to perform the recovery AND the criteria that will be used to evaluate whether it was successful or not

Object Documentation

• DBA specific stored PL/SQL programs
• Day-to-day support scripts (where they are and what they do)
• Monitoring scripts (where they are and what they do)
• Scripts used to perform database administrative changes. I have specific directories that provide output from critical database changes that I have performed and other directories containing the SQL used to make that change
• Operating system scripts. Document what the script does in the beginning of each of your scripts. Ever try to determine what a 400 line script does that was created by someone that knows much more about UNIX scripting than you do? We have all been in that position at one time or another during our career. I’m personally still not good at it. Which is why I never was a great developer (all that IF-THEN-ELSE-OTHERWISE-GOTO-EXIT stuff bored me to tears). Make it easy on your coworkers to understand what the script does by putting comments at the top of the script as well as in the body. Also keep a running history of script changes. What they were and the time they were made

Database Administration Unit Organizational Documentation

• Contact Information
• DBA responsibilities. Which databases and tasks they are responsible for
• DBA unavailablity. Allows application developers to plan for a DBA not being available

It is a good practice to distribute this information to all business units supported by the database administration unit

I hope you enjoyed this blog on documentation and the important role it plays in the Art of Being a Successful DBA.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery