The Usual Suspect

Generally speaking, the majority of database corruption is caused by hardware errors in the disk sub-system. Our database files are sitting on our SAN, or DAS, and are happily going about their lives without any problem until something goes awry. You may not even know there is a problem until it’s too late and a database is marked suspect.

The most important thing to can do, outside of monitoring for these problems, is to keep your storage system’s firmware and drivers up to date and consistent within the system. Sign-up for notifications with your vendor so you will receive these notices, and when you received them, read the little text file that explains what has changed and what will happen if you don’t update. I can’t tell you how often I’ve read through these and run across verbiage like, “previous version can cause data loss.”

These systems aren’t perfect. Stay on top of your firmware and driver updates or you may be turned down when you make that middle of the night support call and need their help.

Automate the Eights

I’ve got a bag of alerts that I like to implement on each SQL Server that I administer. Three of these alerts are for errors 823, 824 & 825.

Error 823

Sometimes referred to as a hard I/O error, this is alerting SQL Server, and you, that a read or write error has occurred. In the case of a read error, Windows has already attempted a retry four times before sending the message. This error can also be captured by alerting on errors with a severity of 24.

This error will be accompanied by the following entry in the SQL Server error log and Windows event log:

“The operating system returned error 823 to SQL Server during a <<MESSAGE>> at offset <<PHYSICAL OFFSET>> in file <<FILE NAME>>. Additional messages in the SQL Server error log and system event log may provide more detail. This is a severe system-level error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online”.

Error 824

Sometimes referred to as a logical consistency error, or soft I/O error, this indicates that Windows was able to read the page from disk but SQL Server has discovered that the page has a problem. This error can be caused when SQL Server determines that there was a checksum error or a torn bits error when it read the page from disk. This is dependent on the page verification option your database is using. This error can also be captured by alerting on errors with a severity of 24.

No matter which occurred, there’s a problem with the page and the following error message will be reported in the SQL Server error log and the Windows event log:

“SQL Server detected a logical consistency-based I/O error: <<ERROR TYPE DESCRIPTION>>. It occurred during a <<Read/Write>> of page <<PAGEID>> in database ID <<DBID>> at offset <<PHYSICAL OFFSET>> in file <<FILE NAME>>. Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.”

Error 825

This error also falls in the hard I/O error category. When this error is received, it indicates that Windows requested a read and the read failed at least one time. Oftentimes, this is the first indication that there’s trouble brewing in the disk sub-system. This error can also be captured by alerting on errors with a severity of 10.

You’ll find the following error message in the SQL Server error log and Windows event log:

“A read of the file <<FILE NAME>> at offset <<PHYSICAL OFFSET>> succeeded after failing<<FAILURE COUNT>> time(s) with error: <<MESSAGE>>. Additional messages in the SQL Server error log and system event log may provide more detail. This error condition threatens database integrity and must be corrected. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.”

Conclusion

You’re going to receive one, or more, of these errors during your career as a Database Administrator. No one is immune. Bad things happen to hardware, and most of the time there’s not a lot you can do to prevent it. However, you can be prepared to respond.

If you receive one of these errors, don’t wait around before you investigate. You’re response is critical to the health of the data you’re sworn to protect. These types of problems don’t go away on their own. Don’t make the mistake of thinking that they will.

So, keep your storage system firmware and drivers up to date, implement these alerts and systematically plan your response. These actions will give you the best chance of a speedy recovery when database corruption strikes.

Scott Caldwell, SQL Server DBA

This meeting is held as soon as the application developers are ready to begin the actual coding process. The ultimate goal of this meeting is for application developers to have a firm understanding of what is required to create and maintain a successful Oracle test environment. Discussions on Oracle utilities, SQL statements, procedural code, Oracle security, batch script testing, benchmarking, testing and monitoring are of prime importance at this time. The DBA must make it clear to all involved that the DBA unit is always available to answer questions and troubleshoot poorly performing SQL throughout the entire application development life cycle. It is imperative that the DBA make every effort to find and correct problems during this stage of the development process. Maintaining a proactive stance instead of a reactive one will always prove to be beneficial when it comes to correcting design problems. There are always enough potential problems lurking in the system for the DBA to solve without adding any additional ones through faulty application design. Topics to be discussed in this meeting include:

• Division of duties between DBA, systems, application development and business units
• Test hardware platform, O/S installation, configuration and administration
• Data feeds required from other architectures
• Data load and update processing and scheduling
• SQL Loader control cards
• Test system availability requirements
• Test system backup and recovery requirements
• Oracle security authorization
• Tools to aid in application development
• Oracle GRID toolset monitoring capabilities
• In-house monitoring tools
• Third-party vendor products
• Benchmarking and performance measurements

Monitoring Performance During Testing

Because this meeting and the previously discussed Setting Up a Successful Test System are closely related, they may be discussed together to save time and provide greater clarity. The DBA and developers need to have an understanding of what information is required to effectively monitor the performance of the Oracle application throughout the entire testing process. The DBA can offer to show developers how to use the various toolsets provided by Oracle (Oracle GRID AWR reports, SQL Trace, explain plan, autotrace, Oracle traces, V$ tables and my old favorite – Statspack) so that developers can play an active role in performance measurement. Suggested discussion topics are as follows:

• Names of all SQL Loader control cards and batch scripts
• Performance measurement goals and expectations
• Determine the test load volume (data volume, concurrent users) required to accurately predict production performance
• Comparison of test system load to estimated production load
• Oracle performance measurement tools. Suggested tools include:
  • Explain
  • SQL*Plus autotrace
  • Oracle SQL trace
  • Statspack
  • Oracle GRID monitoring components
  • V$ performance tables
  • Index utilization monitoring via ALTER INDEX MONITORING statement
  • Third party performance monitoring tools
  • LAN performance monitoring tools
  • Operating system performance monitoring tools

Performance Design Reviews

Information collected from the various performance monitoring tools is discussed at this time. One meeting may be sufficient, but large application development efforts usually require several discussions. If the DBA has maintained good communications throughout the initial stages of application design, there should be few surprises when these meetings are held. SQL tuning and tweaking recommendations are covered in this meeting. Depending on the length of the development process, follow-up meetings can be held to ensure that the application is performing as expected. Some suggested topics include:

• Load testing assessment: Is the load being placed on the test system large enough to accurately predict production performance?
• Review performance statistics collected during the testing process
• Assess SQL coding techniques by reviewing explain plan output for performance critical transactions
  • Index usage
  • Local/join predicates
  • Join methods
  • Subselects
  • View materialization
  • Determine if additional Oracle objects need to be created to enhance SQL performance
    • B-Tree Indexes
    • Bitmap Indexes
    • Function-Based indexes
    • Materialized views
    • Index organized tables
    • External tables

Preparation for Production Turnover

This meeting is held to determine if there are any last-minute questions and to make certain that everyone knows what is expected of them during the final turnover process. All units that have participated in the application design or application design review process are invited to attend. To ensure that all steps necessary for a successful migration to production are executed, the use of a standardized Oracle migration checklist is highly recommended. This document will allow the DBAs and developers to concern themselves with topics that are related to this specific turnover rather than spending time on the more mundane turnover tasks that are just as important, but easily forgotten. Having a complete, well thought-out migration checklist produces a less stressful and less error prone production migration process. Topics include:

• Division of duties between DBA, systems, application development and business units
• Production hardware platform, O/S installation, configuration and operation
• Data feeds required from other architectures
• Data load and update processing and scheduling
• SQL Loader control cards
• Backup and recovery
• Oracle security authorization
• DBA forms and turnover procedures
• Contact information and escalation procedures
• Post-production monitoring tools to be used

Post-production Turnover

This final set of meetings is held after the production turnover is complete. Application logic errors and performance problem resolution are the prime topics of discussion. A comparison of the actual performance to the anticipated performance of the application is also discussed. The review and correction process, by its nature, is iterative. The problems are identified, a plan of attack to solve them is agreed upon and additional meetings are scheduled to review the results. As additional problems are identified, they are added to the list of problems to be solved. This iterative process continues until all major performance issues and application logic errors are addressed. The post production turnover meetings should include discussions on:

• Review performance statistics collected (Oracle GRID, SQL Trace, explain plan, autotrace, Oracle traces, V$ tables and Statspack)
• Assess SQL coding techniques by reviewing explain plan output for transactions experiencing performance problems
• Index usage
• Local/join predicates
• Join methods
• Subselects
• View materialization
• Determine if additional Oracle objects need to be created to enhance SQL performance

Oracle Database Design Review Meetings – Conclusion

These recommendations are not intended to coerce readers into using the Oracle application design review meeting examples verbatim but to emphasize the importance of a structured approach to the design review process. The seemingly endless combinations of application architectures and software products used for application development may require the creation of a fully customized design review process for each application development project. The overall goal of design review meetings is to ensure the involvement of technical support units during the application’s design and implementation. When Oracle design issues are addressed early in the development lifecycle, problems are minimized and the migration from test to production is more easily accomplished.

Check out The Art of Being a Successful DBA – Application Design Review Meetings, Part 1.

Thanks for reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

One of the overlooked duties of an Oracle DBA is to inform, educate and assist the application development staff during the application development process. Although these responsibilities may not be formally assigned to the DBA, the DBA unit often finds that they are providing these services by default. The DBA is often considered to be a seasoned veteran who spends most of their time learning the subtle eccentricities of the database management system.

It is the DBA’s responsibility to ensure that the overall design and implementation of the new application is proceeding according to expectations. Although application developers may be experts in SQL, procedural languages (Java, PL/SQL, C variations, TSQL, etc.), they still turn to the DBA for help with standards, procedures, performance design, error recovery and complex SQL.

A continuous and effective dialogue between the DBA unit, system support personnel and application developers is of utmost importance throughout the entire application design and implementation process. One method to foster effective communications is for the DBA unit to create a series of new application design review meetings. These meetings can be organized in a manner that promotes efficient application usage of the Oracle database environment. The design review meetings can be scheduled during logical break points between the different stages of the application development process.

The database administration team should invite representatives from the development and business units to assist in the creation (and enforcement) of the standardized review process. Application development teams will be able to help tailor the design review process to satisfy their specific design and implementation needs. Business users who will be impacted by the production turnover will provide valuable input on implementing new applications for their specific business areas. Customized checklists are created to ensure that all application and business specific issues are addressed during each meeting of the design review process.

It is recommended that the database administration team then communicate to all application development and business areas that any internal applications created without following the standardized review process will not be migrated to production. Having the application teams and business units participate during the creation of the standardized review process allows the DBA team to enforce the policy without being viewed as dictatorial or making rules in a vacuum. In addition, every effort should be made to “sell” the standardized design review process in all communications to application development teams and business units.

The following list of meetings can be used as a starting point in the creation of a structured application design review process:

Initial Overview of Proposed System

The kickoff meeting is held with datacenter operations support managers, operating system administrators, application team leaders, system analysts, end-user management and data administrators to discuss the application’s general purpose and function. This information will allow support technicians to determine the impact the new application will have on existing systems and allow them to begin planning the application design implementation process. The following information should be covered in this meeting:

• General description of purpose and function
• Application scope (enterprise wide application that affects all business units, intra-departmental that affects several business units or departmental)
• General application size (estimated number of programs, data objects)
• Criticalness of the application (mission critical, application critical, non-critical)
• Application availability requirements and downtime windows
• Application type (decision support, business intelligence, data warehouse, data mart, online transaction processing)
• Architecture design (web server, application server, N-tier, distributed database environment)
• Advanced features required (replication, advanced queuing)
• Data feeds required from other architectures
• Load processing
• Batch processing
• Online transaction processing
• Development tools used to build front-end application screens
• Third-party tools used to provide Ad-Hoc query access
• Procedural language used to build business logic (Java, PL/SQL)
• Application development methodology to be used (Agile, Waterfall, RAD)
• Number of concurrent users
• Disk storage and data growth estimates
• Highly available architecture discussion (RAC, Oracle Fail Safe, Data Guard, hardware vendor clustering and failover)
• Performance expectations
• Criteria used to judge performance
• Security and auditing requirements
• Hardware platform, O/S preferences/selection and sizing discussion
• Hardware platform, O/S installation, operation and administration
• Division of duties between the DBA, application development and business units

Logical Data Model Review

This meeting is convened as soon as the logical data modeling effort is finished. The major emphasis of this meeting is to determine if the logical data model is complete and correct. The application’s process model (if one is available) can also be verified at this time. Volume statistics, object growth rates, purge criteria, referential integrity needs and application-naming conventions are also discussed. Knowing your data before hand is essential to designing processes to manipulate that data. The following topics are covered in this meeting:

• Determine if the data model is fully documented (entities, attributes, relationships)
• Attributes have correct datatype, length, NULL status, default values
• General discussion of business rules that are to be enforced by database level constraints
• Not null constraints
• Check constraints
• Unique constraints
• Primary key constraints
• Foreign key constraints
• Business rules to be enforced by triggers and procedures
• Business rules to be enforced by application code
• Logical/process model comparison
• Volume statistics
• Growth rates and purge criteria

Designing for Performance

This meeting is held with the application development units before any physical DDL is generated by the DBA. Proper transaction design and efficient SQL coding results in less performance-oriented database alterations made during the latter stages of the design and implementation process. Determining predicate usage will allow the DBA to create additional database objects (indexes, materialized views, index organized tables) to increase SQL, and subsequently, application performance. The following information is discussed:

• Normalization vs denormalization or “lets collapse those 49 tables into one big one for fast read access”
• Table access requirements and predicate usage
• Database objects used to increase SQL performance including:
  • B-Tree Indexes
  • Bitmap Indexes
  • Function-Based indexes
  • Materialized views
  • Index organized tables
  • External tables
  • Data partitioning algorithms (range, hash, list, etc.)
• Process parallelism (parallel query, parallel DML, parallel load)
• Transaction modeling
• Oracle SQL performance features
• Full table scan vs index access
• Hash joins
• Star joins
• Index skip scans
• Index fast full scans
• Cursor sharing (SQL statement reuse and soft parse vs hard parse)
• Bind variables
• Reinforcement of effective SQL coding techniques

Read The Art of Being a Successful DBA – Application Design Review Meetings, Part 2.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Following proper naming conventions is absolutely critical to us as a remote DBA services provider. We have dozens of DBAs on staff now that are responsible for supporting hundreds (and hundreds) of different database ecosystems. To ensure DBAs can seamlessly tranfer from one environment to another, we must establish standard naming conventions and document everything in our customer’s environments.

Each one of our DBAs can go to any platform and quickly navigate to the script and monitoring tool directories, find the script they are looking for and execute it. I feel that this is so important to the quality of support we provide that we have selected a group of technicians that are responsible for creating, adjusting and auditing naming conventions here at RDBAE. These folks, led by Doug Russel, our Proactive Monitoring Center Team Lead, are responsible for laying the groundwork to ensure that our naming conventions and best practices are being followed.

I’ll continue our discussion on naming conventions by providing you with a few sample recommendations. As always, these recommendations are not intended to be all-inclusive. They are a few examples to help start you in the right direction.

Follow OFA Naming Conventions
OFA stands for Optimal Flexible Architecture. The OFA standard is a set of naming conventions and configuration guidelines that:

• Distributes I/O among different disk drives.
• Facilitates ease of administration by creating naming conventions for mountpoints, directories, file suffixes, database binary directories and database output (i.e, background dump, core dump).
• Improves the DBA’s ability to manage and administer database growth.

These standard naming conventions and placement guidelines are intended to improve database performance by spreading I/O, protect against drive failures and also allow administrators to more easily assume the responsibility of supporting new database environments. In addition, because OFA standards are well documented by the Oracle Corporation, newly hired DBA and consultants are able to more quickly assume administration responsibilities.  To learn more about OFA, do a Google search on the key words “Optimal Flexible Architecture”.  You will find all of the documentation you need!

Create and Standardize Monitoring and Administration Scripts
All DBAs have their own set of favorite Oracle administration and monitoring scripts. They find them on the web, get them from third-party books and trade them like bubble-gum cards. They then tune, tweak and tailor them until they fit their own unique style of administration.  It is highly recommended that database administration units create a set of scripts for daily administration/monitoring, hot and cold backups (RMAN or OS), loads, exports/DataPump and common administrative activities. This library of scripts can then be installed on each server administered by the team. Personalizing the scripts should be highly discouraged. Any modifications to the library can be reviewed during team meetings.

Assign a team member the responsibility of being the script library owner. The script library owner will be responsible for keeping track of all script modifications and ensuring a rigorous review and test plan is executed on the new scripts before they are installed on the supported servers.   Monitoring scripts are even more crtical. They are the foundation of the DBA’s support activities. If your monitoring scripts aren’t working, you are failing in your role as “protector of the organization’s key data assets”.  Here at RDBAE, we have a virtual team that is assigned the responsibility of keeping track of literally dozens, and dozens, of Groundwork monitoring plugins. These technicians use an off-the-shelf code library with full check in/check out features and review capabilities (for approval before release).

All scripts and directories should have names that allow them to be easily recognized. It irks me to no end when I log in to a server, navigate to a script directory and find files named SQL1.SQL, SQL2.SQL, FIXIT.SQL, TUNEIT.SQL… Fix what? Tune what? What the?  Personally, I would prefer to have a SQL file name that is long, but descriptive, as opposed to a name that doesn’t provide me with any clue on its contents.

It is also important to follow a strict naming convention for your output files. Whether the output is from a SQL, DDL or operating system command, the output file should be placed in the appropriate directory and have a descriptive name. The file name should also contain the date the output was created.

For example:
create_mcfinemp1_table_012609.out would be the output from the DDL statement execution that created the MCFINEMP1 table on January 26, 2009.

Object Naming Conventions
This is the area where having a Data Administrator available is certainly beneficial. These are the folks that are trained in naming convention best practices. If you don’t have the good fortune of having a DA available, it will be up to you and your application developer counterparts to create a set of naming convention guidelines. It is important to create a naming convention that contains abbreviation guidelines that are used to shorten object names.

Here’s an example of an index naming convention I have used in the past:

INDEX
Definition: An index is an object that ensures efficient access to the data by using an ordered set of pointers to the table’s data rows.  Indexes can also be used as the physical mechansim to ensure key value uniqueness.
Naming Conventions:
Format #1   -   tb_idx_uu
tb – table name (abbreviated if required)
idx – constant identifying this object as an index
uu – ascending numeric or character index identifier (used to identify multiple indexes per base table)
Example: MCFINEMP_IDX_01 would be the first index on the MCFINEMP table.

There are dozens of guidelines available on the web. In addition, many of the books on Data Administration provide suggested naming conventions. We also have our own personal preferences. For example, many DBAs like to include column name abbreviations in their index names.

What the standard naming conventions are is not as important as creating them and adhering to them religiously. That is the key to success.  These recommendations are not intended to coerce readers into using the provided naming convention examples verbatim but to emphasize the importance of a creating a set of standardized naming conventions and then strictly adhering to them.

Thanks for reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Documenting processes, procedures and best practices is a task that is often considered to be boring and mundane. Most DBAs would rather perform virtually any other activity than sit in front of a screen using a word processor. As a result, creating documentation is often postponed until the DBA has a little free time to kill. Today’s database administration units are operating with smaller staffs, tighter budgets and ever-increasing workloads. The end result is that the documentation is either never created or created and not kept current.

But a robust detailed documentation library creates an environment that is less complex, less error-prone, reduces the amount of time DBAs spend learning new database environments and reduces the overall time spent on day-to-day support activities. DBAs are able to spend more time administering the environment than finding the objects they are trying to support and the processes and programs used to administer them.

The nature of my business as a remote services provider demands excellent documentation. The majority of environments we administer weren’t designed by my organization. The only way that we can ensure high quality and high-speed administration of these environments is to document them thoroughly. We document everything, from initial connectivity and customer contact sheets to detailed information on batch job streams and individual program execution. If we need to be aware of it, we have it documented.   We have dozens (and dozens) of internal best practices also documented.   If you are evaluating remote database services providers, you need to ask them for examples.

Documentation is also the foundation of many of the other disciplines I will be discussing in future blogs. Let’s continue our discussion with a few helpful hints to get you started.

Understanding the Important Role Good Documentation Plays
We all generally understand the benefits that documentation provides. I think that all readers will see the importance I personally place on documentation in upcoming blogs. Virtually every topic that I will discuss, from repeatable processes to paranoid administration best practices, I will be mentioning what to document and the important role that documentation plays.

Let me reaffirm my opinion in this one sentence:

Documentation is one of the foundations for virtually everything I will be discussing in this series on The Non-Technical Art of Being a Successful DBA. Enough said.

Creating an Organizational Environment That Fosters Good Documentation
I have been leading database teams off and on for 15 years now.  It is my responsibility as manager to create an environment that fosters the production of robust and high quality documentation. Let me describe some of the challenges that I have faced in the past and how I have overcome them.

Since I view high quality documentation to be my responsibility as the director of database services here at Remote DBA Experts, I ensure that it becomes part and parcel of every DBA’s job description.    If we feel that a candidate DBA doesn’t value the importance of documentation, we don’t hire them.    It’s OK if they don’t like to do it, they just need to express that it is something that is required to create a high-quality support environment.

I will add time for documentation when I estimate the amount of time it will take me to perform an administrative task during project planning meetings. I don’t settle for “we can do that after the project is complete” as an answer from an application project manager.

If I get overruled and lose one battle, I still make sure I document my activities after the project’s critical path is complete. And I don’t let a single setback prevent me from telling everyone in the next planning meeting that I need so many hours to document what I have done.

If you continuously sell the importance of documentation sooner or later you will begin to wear your opponent’s down. Although I prefer to call it “being relentless”, I’m sure that many of the application development managers (and my own managers) viewed it as “being a ….” (insert your favorite description here).

Every document I have created that provides a list of activities I , or my unit, needs to perform during a project has documentation included. It helps to integrate it into the fabric and culture of my organization’s environment.

Making Documentation Second Nature
You also need to ensure that generating documentation becomes a natural part of your daily activities. You must continuously remind yourself that documentation is a primary and integral part of providing high quality support services to your customers.

You must also remind yourself that it makes your job easier and benefits your fellow DBAs. You know what has really irritated me the most during my career? When a fellow DBA needs to be out of the office for a time and asks me to “help them out” by performing a complex, application specific administrative activity and then tries to verbally tell me how to perform the 300 steps it takes to do it.

Ever try to refresh an ERP application test environment from production when that test environment doesn’t have enough space to hold all of production’s data? 4,000 steps later and you begin to second-guess your choice of professions. That was the exact request from one of my fellow DBAs a few years ago when I worked for a large aluminum producer here in Pittsburgh. Not only did he get me to do the refresh but I also had to document the process for him along the way. Some call that being a good coworker; I would view that as having a big sucker taped to my forehead. Live and learn, I guess.

The moral of this story is: If you don’t want to be the only one that can perform that 900 step ERP Application production to test refresh, document it! If you don’t want to be called by the on-call DBA because he doesn’t know exactly where to add a datafile in an emergency situation (like an application developer forgetting to tell youu that they were loading 10 million additional rows into that 100 row table), document it! The more you document, the easier your life as a DBA becomes.

I’ve never had a great memory. It makes documentation easy for me. I also like to write and that helps. But I will admit that there are times that I would rather perform virtually any other activity than document.

But it has become easier because I continuously reaffirm to myself the importance of documentation. The more you reinforce that to yourself, the more second nature (and easier) it becomes.

Making Documentation Easy
I’m a huge fan of documentation templates. Here at Remote DBA Experts we have, or are in the process of creating, templates for everything we document. We have templates for documenting connections to our customer’s environments, their backup and recovery environments, their application specific processes to name a few. If it needs to be documented on a regular basis, we have a template for it. We also have generic templates for documenting environments and activities that don’t fit into other templates.

Microsoft Word document templates provide many features that streamline the documentation process and help to improve the quality of the content they store. I try to take advantage of as many features as I can. I use drop down selection menus, check boxes and radio push buttons to improve the speed and quality of the documentation process. I also take advantage of the help pop-up feature that Microsoft Word provides to create a detailed description of what information is to be entered into that field, check box or radio button.

I created my first template close to 15 years ago. It wasn’t created to document a process, but to document a change request. I’ll be covering change management in an upcoming series in this blog. Was the document crude? You bet. But it did the job. It has gone through numerous iterations and migrations from one Word release to the next. I continue to use it at organizations that don’t have a formalized and documented change management process. And you can’t believe how many I have come across that don’t.

There are dozens of software companies that offer content management solutions. Database vendors have also recognized this as a lucrative market. All of the major database vendors now offer advanced content management software, each one trying to outdo the other in the number of bells and whistles that their product’s offer.

Do a quick search on Google for documentation content management software and you will find out just how many competing products there are.

Content management products offer check-in/check-out features, document versioning, web portal access and advanced workflow capabilities to name just a few of the features desgined to improve content management. The competition in the content management market space is fierce to say the least. Content management vendors know that continuously adding new bells and whistles to their products is not just important for increasing market share; it is critical for their survival. Product costs can range from thousands to tens of thousands of dollars (or more).

If you have the funds and your management understands the benefits that a full-blown content management package provides, by all means begin a content management product analysis. But if you don’t have the funds, create a shared drive on your network and declare it to be the “DBA Documentation Portal.” We will be using a combination of Microsoft Sharepoint and Sales Force as our own DBA Documentation Portal.  Sales Force’s interaction with our document libraries provides us with a highly secured, highly customizable document storage environment.  It is our “one stop shop” for customer information.

What to Document
By all means this is not an all-inclusive list of what can be documented. Consider it as a starter kit to help you begin your quest for “documentis nirvanas”. Is some of this overkill for your particular environment? Maybe, but just consider this a general, high-level list.

Database Environment Documentation

• Naming conventions
• Servers (server names, operating system release, hardware vendor)
• Databases (vendor, database version, features enabled)
• Directories (everything from Oracle software installations that don’t follow Oracle standards, application code storage to personal script locations). If you, or a fellow DBA needs to find it, document where it is!

Application Specific Documentation

• Application type (i.e. data warehouse, online transaction processing, decision support, third-party application name and functionality it provides).
• Business unit requirements and related information for supported databases
• Uptime requirements (i.e. 24 X 7, 8 X 5)
• Database downtime windows
• Critical job processes
• Business unit and application developer contact lists
• Turnover windows for database changes
• Problem notification and escalation procedures
• Security Sensitivity. How sensitive is the data?

Process Documentation

• Repeatable administrative processes (covered in an upcoming blog)
• Backups – Probably the most critical set of documentation you will ever create. Document how it is backed up, what scripts back it up, where the back up is going to, tape retentions and backup message directories. If it is involved with a backup, DOCUMENT IT. Review the document with other units that are involved in the backup and recovery process. It is your responsibility to ensure that you don’t hear an operator say “What retention period? Nobody told me we were to have a retention on these tapes.” when you are in a recovery situation. Remember that Oracle states that human error, including miscommunications, is responsible for over 90% of failed recoveries. If you want to reduce recovery failures DOCUMENT THE PROCESS AND REVIEW IT.
• Oracle utilities. Document Oracle Import/Export input and output directories and where the files are that execute the utilities. The same with Oracle’s load utility. Document the execution scripts and input and output directories. A short note on why you are running it is also required
• Anything else you run on a regular basis to support a specific application
• Change Management. I’ll be spending an entire blog, or two, on this
• A daily monitoring activity checklist to ensure that no activity is missed. We have daily, weekly and monthly activities that are to be performed for each of our customers
• Assuming on-call responsibilities.  You need to make sure you are prepared for the on-call process.   The checklist should include what other DBAs are working on during your on call rotation, customer outages taking place, connecting to environments that are hard to connect to, etc..
• Complex administrative activities performed regularly
• Test and reporting database refreshes
• Data reorganizations
• Disaster recovery tests. The processes required to perform the recovery AND the criteria that will be used to evaluate whether it was successful or not

Object Documentation

• DBA specific stored PL/SQL programs
• Day-to-day support scripts (where they are and what they do)
• Monitoring scripts (where they are and what they do)
• Scripts used to perform database administrative changes. I have specific directories that provide output from critical database changes that I have performed and other directories containing the SQL used to make that change
• Operating system scripts. Document what the script does in the beginning of each of your scripts. Ever try to determine what a 400 line script does that was created by someone that knows much more about UNIX scripting than you do? We have all been in that position at one time or another during our career. I’m personally still not good at it. Which is why I never was a great developer (all that IF-THEN-ELSE-OTHERWISE-GOTO-EXIT stuff bored me to tears). Make it easy on your coworkers to understand what the script does by putting comments at the top of the script as well as in the body. Also keep a running history of script changes. What they were and the time they were made

Database Administration Unit Organizational Documentation

• Contact Information
• DBA responsibilities. Which databases and tasks they are responsible for
• DBA unavailablity. Allows application developers to plan for a DBA not being available

It is a good practice to distribute this information to all business units supported by the database administration unit

I hope you enjoyed this blog on documentation and the important role it plays in the Art of Being a Successful DBA.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

I’ve been able to learn from my mistakes and make changes in my technical and non-technical approach to database administration to overcome my shortcomings. I’d like to cover some of my personal best practices that have made my life as a DBA more productive and my customers and co-workers happier.

I’d like to take a minute to describe some of the jobs I’ve held and how they have taught me to learn the value of becoming proficient at both the technical and non-technical disciplines required to become a successful DBA. I’ll continue this week’s discussion with what I think is probably the most beneficial set of recommendations I could provide to you as readers. Some of the information is certainly “intuitively obvious” as my old boss used to say but it never hurts to have a refresher. In upcoming blogs, I’ll remove the personal stuff and get down to the business of providing you with information on The Non-Technical Art of Being a Successful DBA. We will then turn our attention to the wonderful world of Oracle and SQL Server database administration Tips, Tricks and Techniques.

There are numerous jobs where you have to be on top of your game at all times. The medical profession and police work come to mind when I think about folks that have to be “mistake free” to keep us all safe and healthy. I often remind myself that the stress I let my job generate for me is nothing compared to what these aforementioned folks must experience.

As Oracle database administration goes, being an Oracle Instructor is about as unforgiving as it gets. You get dropped into a classroom that has twenty PCs running Oracle and twenty students depending upon you to teach them. My first attempts at teaching a dozen years or so ago could be described as “less than stellar.” I had a review from my first class state “this was the most boring person I have ever met.” I felt bad about that until my senior instructor showed me one of his first reviews. It had in big red letters at the top of the page “I HATED YOU.” I felt better after reading that.

I must admit I was a little timid when I started, but those students that attended later classes know that I decided to let my own bombastic personality show through. Let’s say that none of my subsequent classes were ever boring after that.

After that less than glorious start, I worked hard to become a better instructor, eventually winning one of Oracle’s top instructor awards with a 95% student approval rating. How I ever ended up in a teaching profession where the students graded the teacher, I’ll never know. Just as a joke I started one class by stating “Oracle has instituted a new procedure that will allow me to grade your participation and learning skills and send them to your employers.” The look on everyone’s faces was priceless. I still view being asked by Oracle to certify fellow instructors as one the greatest achievements of my career. I took that job very seriously.

I can tell you one thing about Oracle DBA students, they are a rough bunch. I started a DBA I class by blowing up a database create and couldn’t get it to work correctly after a second attempt. This after doing it successfully hundreds of times. A corrupt drive on my PC was the culprit. After turning around and seeing the “who is this idiot” look on my student’s faces, I suddenly felt like a wounded fish in the middle of a pack of hungry piranha.

Oracle instructors describe that problem with a single phrase “losing creditability with the class.” You did that and you were lunchmeat. But I never let that potential loss of creditability prevent me from saying “I don’t know, I’ll get back to you.” Or I would look up the information for the student on the spot. That provided two benefits: answering the question and showing students the importance of knowing where to look for the answers.

Assisting a class of twenty students through backup and recovery labs certainly helped my debugging skills. If you ever want to find out how bullet proof an Oracle database can be, watch 20 installations being abused by students in a backup and recovery class. I always started my backup and recovery class with this statement “The fastest way to lose your job in this profession is to lose data for your company. You can be a Tom Kyte and a Jonathan Lewis X 2, but if you can’t recover a database, you aren’t of any use to your employer.” That always seemed to ensure that they paid attention during the remainder of the class. That was one class where I absolutely pounded information and best practices into their collective heads. We were all worn out by Friday.

I enjoyed teaching and became friends with each and every group of students I taught. I always felt sad when we parted and always worried that I missed that one tidbit of information that could help them.

On-site consulting is also a tough profession. The customer brings you into an environment that you know nothing (or little) about and expects you to be productive and solve their problems – immediately. That is as it should be. They pay a premium and expect good service. It is the consultant’s fault if that doesn’t happen. I liked that job because it was very challenging and forced me to continuously refine my skill sets and talents to keep my customers happy.

But I have had my fair share of sweaty palms at many a client site. I didn’t have the luxury of being an Oracle consultant. They always seemed to have an endless stream of people to draw from for support. I usually ended up in a wiring closet with a manual in one hand and a keyboard in the other. I actually spent three weeks in a wiring closet at one client’s site.

Remote database administration also requires excellent technical and non-technical skills. You must continuously strive to keep your customers happy, all of which hold you to a very high standard.  You must expect that from all of your customers in this profession.   They have handed the keys to their most critical data assets to your firm.   Trust me when I say we at Remote Database Experts don’t take this responsibility lightly.

If you want to be at the top of your game, become a DBA for a remote database administration firm.  Describing competition in the remote DBA industry as fierce is like describing the Titanic as “having a small leak.   It is a very challenging job.

You become successful as a remote DBA services company by providing a superior level of support to your customers. As a technician, it’s a great job! You can be installing a RAC/LINUX environment in the morning and be debugging an Oracle Applications concurrent manager problem in the afternoon. From warehouses to HUGE web databases having thousands of concurrent users and from manufacturing to financials, you work on it all.

The Art of Being a Good DBA
I started off this blog describing some of my jobs not to impress you with my background but to hopefully help you understand how important it is to me to provide you with this first series of non-technical recommendations. These jobs quickly laid bare every hole in both my technical and non-technical approach to database administration. They not only laid them bare but also amplified their negative results. I found that becoming lax in the non-technical areas of database administration and not following my own best practices could create just as much havoc as a technical problem. As I have stated numerous times, some of us are brilliant and some of us are lucky. The rest of us have to be thorough.

To ensure that I did not allow a failure in non-technical areas to complicate my life as a DBA, I needed to learn how to document, communicate and follow my own best practices. I learned the tricks of the trade to reduce the amount of time it took me to administer my environments and ensure I didn’t get paged at night. I found that I could trust some author’s information and not others. I learned that by keeping my recovery skills sharp, I reduced my level of anxiety when I got one of those two in the morning “I think we just formatted one of your Oracle disks by mistake” calls (actually the caller stated they accidentally formatted the entire array consisting of dozens of disks by mistake). The list goes on and on…

To continue, here’s a laundry list of what will be covered in upcoming blogs:

• Why good documentation is so essential and documentation best practices
• Naming conventions
• The benefits of Data Administration
• The key to successful application implementation – Design Review Meetings
• Improving the speed and quality of database administration by keeping your environments organized and uncluttered
• Creating and maintaining tips, tricks and problem resolution libraries
• Script foraging
• Reducing stress and preventing failed recoveries by keeping recovery skills sharp
• The benefits of repeatable processes. If it worked that way once, it’ll probably work that way again
• Proceduralizing the change management and production turnover process
• Time Management and the art of project and process time estimation
• Maintaining good relationships with customers, end-users and fellow technical personnel
• The Foot Rule of Thumb
• Knowing where to look when you don’t know the answer (reprise from earlier blog)
• Using Service Level Agreements to ensure your internal and external customers know what to expect from you
• Monitoring by day so you don’t get called at night
• Monitoring space growth and performance statistics historically
• Paranoid administration best practices. Many of which are stolen from probably the most paranoid (and error free) DBA I have ever met. Jeff “I never met a problem prevention method I didn’t like” Kondas.
• Preventing errors from reoccurring by creating and following Corrective Action Procedures

Some of the topics will require an entire blog to cover while other blogs may group many topics together. The intent of this series of upcoming blogs is to help beginners create an approach to improving in non-technical areas and act as a refresher to those of us that have been in this profession for a while. Some of these recommendations come from problems that have occurred as a result of me not placing the same emphasis on non-technical areas as I did the technical ones. Many of the recommendations come from my experiences working as a mainframe DBA that, as we all know, is a very structured environment that demands rigorous adherence to best practices. The remainder comes from my experiences working as an educator, consultant and services provider.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery