SQL Server File Initialization

The SQL Server storage engine utilizes an initialization process on data and log files. This process fills the files with zeros in order to overwrite any existing data that may be on disk. The initialization process occurs when you create a database, restore a database or file group, add files to an existing database or increase the size of an existing database file. These disk operations are blocking processes that will cause SQL Server to wait until they are complete.

What is Instant File Initialization?

Simply put, instant file Initialization removes the file initialization penalty by negating the need to fill the newly allocated space with zeros before it can be used. This is incredibly important and can reduce disk allocation times from minutes to milliseconds. This is incredibly important when you think about restoring a VLDB and how long you’ll have to wait just for the empty database files to be initialized.

There are a couple of caveats to mention. The first is that instant file initialization is only available on Windows Server 2003 or later. If you’re on Windows Server 2000 then you’re out of luck. The second is that this only works for database data files and is not available for transaction log allocations. The last thing to note is that this is not available if you’re using Transparent Data Encryption (TDE) in SQL Server.

Am I Using Instant File Initialization?

If you don’t know if you’re using this capability then I recommend you find out as soon as possible. Instant file initialization is only working for you if the SQL Server service account has been granted SE_MANAGE_VOLUME_NAME .

Figuring this out from the comfort of SSMS turns out to be pretty simple but it does require the use of xp_cmdshell in order for the query I’ve provided to work. If your security policies restrict the use of xp_cmdshell then you’ll have to figure this out the hard way or turn on xp_cmdshell before you execute and then turn it off after you’re done. If xp_cmdshell is enabled then execute the following script to see if you’re receiving the benefit:

EXEC xp_cmdshell ‘whoami /priv’;

GO

What have we just done? Well, we’ve just executed the Windows “whoami” command in the context of our database service account. We included the /priv parameter because otherwise this tool returns quite a bit of information and we only need to see the security privileges. It’s important to note that the output will return all the security privileges whether they are granted to the account or not. The key is to look to the right of the privilege to see whether is disabled or enabled. Have a look at the output below:

If you scan down to the bottom of the list you’ll find the SE_MANAGE_VOLUME_PRIVILEGE near the bottom. You’ll see that to the right the privilege it is listed as enabled. This is great news and we’re not paying the initialization penalty during data file allocations.

Wait, Instant File Initialization is not Enabled!

If you executed the query and came to the unfortunate conclusion that instant file initialization is not enabled, don’t worry because it’s very easy to implement.

First off, open up the “Local Policy Editor” on the database server and navigate to the “User Rights Assignment” node under the “Local Policies” node. When you select the “User Rights Assignment” node all of the policies will be displayed in the pane at the right.

You’ll need to scroll down to find policy. It’s named “Perform volume maintenance tasks.” You’ll notice that the users or groups that have been granted this privilege are listed to the right. Open the policy and add the SQL Server service account.

Unfortunately, once this is complete you’re still not finished. You’ll need to restart the SQL Server service for it to take effect.

Conclusion

I think it’s easy to see how this feature will benefit your environment in a number of situations. So, take the time today to check this out and save yourself all the waiting when time matters most.

Scott Caldwell, SQL Server DBA

You’ll find references regarding these in blogs across the Internet and some are more useful than others. Obviously, these are unsupported commands and should only be used with great care.

TRACE FLAGS

There are quite a few undocumented trace flags in SQL Server.  However, I’m only going to touch on four today. They are trace flags 3004, 3014 3604 and 3605.

These can be enabled for the current session with the following commands:

– Enable the trace flags

DBCC TRACEON(3004);

DBCC TRACEON(3014);

DBCC TRACEON(3604);

DBCC TRACEON(3605);

GO

These trace flags are going to be used in conjunction with a database restore so we can get an idea about what SQL Server is doing during the process. We’ll be looking at output in the message output screen and output sent to the SQL Server error log.

Before we go much further, I want to briefly explain the purpose of these flags and why it’s important to use them together to get the information we’re looking for.

Trace Flag 3004

Most Database Administrators are aware of instant file initialization. In a nutshell, when instant file initialization is enabled the data files do not need to be zeroed out during creation. This can save an incredible amount of time during the restoration of VLDBs. As you can imagine, the zeroing out of a 1 TB data file can take a very long time.

Trace flag 3004 turns on information regarding instant file initialization. Enabling this trace flag will not make this information available to view. You will still need to turn on trace flag 3605 to send this information to the error log.

Trace Flag 3014

Trace flag 3014 provides detailed information regarding the steps performed during the backup and restore process. Normally, SQL Server only provides a limited amount of information in the error log regarding these processes. By enabling this trace flag you’ll be able to see some very detailed and interesting information.

Trace Flag 3604

Trace flag 3604 can be used under a variety of circumstances. If you’ve ever used DBCC IND or DBCC PAGE then you’ve probably already used trace flag 3604. It simply informs SQL Server to send some DBCC output information to the screen instead of the error log. In many cases, you have to use this trace flag to see any output at all.

Trace Flag 3605

Trace flag 3605 will send some DBCC output to the error log. This trace flag needs to be enabled to see the instant file initialization information made available by trace flag 3004.

Restore without Trace Flags

I created a full backup of the AdventureWorks database and then restored it with the following command:

– Restore the database

RESTORE DATABASE AdventureWorks

FROM DISK = ‘C:\TEMP\ADVENTUREWORKS.BAK’

GO

Without these trace flags the following information is returned to the message output screen:

You can also see very similar information in the error log. I returned the contents of the error log with the following command:

– Read the error log

EXEC xp_readerrorlog;

GO

Restore with Trace Flags

I’m going to perform the restore again. This time my four trace flags have been enabled and I’m hoping to see some additional information in both the message output window and the error log.

In order to make sure the error log is easy to read, I’ve cycled it with the following command:

– Cycle the error log

EXEC sp_cycle_errorlog;

GO

The next step is to execute the restore again.

– Restore the database

RESTORE DATABASE AdventureWorks

FROM DISK = ‘C:\TEMP\ADVENTUREWORKS.BAK’

GO

After the restore is finished, and with the log cleared and the trace flags enabled, we get the following information in the message output window:

It’s easy to see that this output is far more detailed when compared to what we received during our last restore. You notice that there are quite a few additional steps that were not reported when the trace flags were not enabled.

But wait, we also need to check the error log to see what additional information is available there too.

– Read the error log

EXEC xp_readerrorlog;

GO

The most interesting information in the error log is in regard to instant file initialization. You can see that SQL Server is zeroing out the data file during the container prepare process. This means we’re not benefiting from instant file initialization.

Conclusion

It’s important to understand that SQL Server purposely hides this level of detail from us on a daily basis. Arguably, this information isn’t really that necessary. However, if you feel as though you need to have a greater understanding about what’s going on inside the database engine then this is a good starting point on your journey.

Scott Caldwell, SQL Server DBA

During this time of data marts, pivot tables made from excel spreadsheets, and data spread throughout the company, this concept has made a big comeback.  Mergers and acquisitions, as well as data being managed by separate departments, contribute to the lack of data consistency across the organization. For example, you would never have your current cable provider calling to inquire if you’d like to switch your cable to… your current cable provider.  This type of mishap usually occurs because there is a ‘sales lead’ database that is not reconciled with a ‘current customer’ database.  Master Data Management (MDM) requires the company to first define their data and agree upon what it all means.

The objective of MDM is to ensure that all of the non-transactional data in the company (what some might call a reporting database or a data warehouse) is consistent throughout the organization.  In other words, the idea is that the ‘right hand’ of the organization will always know what the ‘left hand’ is doing.  Once the data definition is agreed upon, the company must ensure that all data collection and data manipulation follow the data definition perfectly.  The data is then processed and stored in a single location, and all BI, reporting, CRM, ERP, and other functionality feed off of this single repository.

Virtual Master Data Management (VMDM) is considered the fourth generation of MDM solutions.  VMDM uses abstraction layers in the data model to create a metadata catalog.  The data is not actually consolidated into a single storage location, like it is in the more typical MDM solution.  Instead, the catalog contains an index of the information and is used to make sure that all data is consistent across the organization and that the ‘true’ source of each piece of information is known and documented.  The data is dynamically transformed when requested by a BI, CRM, or ERP solution.

There is an obvious time advantage in using VMDM as opposed to a typical MDM solution.  The project life cycle of MDM includes the definition and analysis phase, as well as the development phase, during which ETL processes are written to transform and load the data into the newly defined structures.  Since you are often storing the data both in its originating system, and also in the MDM solution, you can save on storage costs with VMDM.

Although MDM has many positive attributes, many companies have found it cumbersome to implement in its purest form.  The concept of VMDM allows for data governance with a significant time and cost reduction.

Katy Park, SQL Server Team Lead

The Usual Suspect

Generally speaking, the majority of database corruption is caused by hardware errors in the disk sub-system. Our database files are sitting on our SAN, or DAS, and are happily going about their lives without any problem until something goes awry. You may not even know there is a problem until it’s too late and a database is marked suspect.

The most important thing to can do, outside of monitoring for these problems, is to keep your storage system’s firmware and drivers up to date and consistent within the system. Sign-up for notifications with your vendor so you will receive these notices, and when you received them, read the little text file that explains what has changed and what will happen if you don’t update. I can’t tell you how often I’ve read through these and run across verbiage like, “previous version can cause data loss.”

These systems aren’t perfect. Stay on top of your firmware and driver updates or you may be turned down when you make that middle of the night support call and need their help.

Automate the Eights

I’ve got a bag of alerts that I like to implement on each SQL Server that I administer. Three of these alerts are for errors 823, 824 & 825.

Error 823

Sometimes referred to as a hard I/O error, this is alerting SQL Server, and you, that a read or write error has occurred. In the case of a read error, Windows has already attempted a retry four times before sending the message. This error can also be captured by alerting on errors with a severity of 24.

This error will be accompanied by the following entry in the SQL Server error log and Windows event log:

“The operating system returned error 823 to SQL Server during a <<MESSAGE>> at offset <<PHYSICAL OFFSET>> in file <<FILE NAME>>. Additional messages in the SQL Server error log and system event log may provide more detail. This is a severe system-level error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online”.

Error 824

Sometimes referred to as a logical consistency error, or soft I/O error, this indicates that Windows was able to read the page from disk but SQL Server has discovered that the page has a problem. This error can be caused when SQL Server determines that there was a checksum error or a torn bits error when it read the page from disk. This is dependent on the page verification option your database is using. This error can also be captured by alerting on errors with a severity of 24.

No matter which occurred, there’s a problem with the page and the following error message will be reported in the SQL Server error log and the Windows event log:

“SQL Server detected a logical consistency-based I/O error: <<ERROR TYPE DESCRIPTION>>. It occurred during a <<Read/Write>> of page <<PAGEID>> in database ID <<DBID>> at offset <<PHYSICAL OFFSET>> in file <<FILE NAME>>. Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.”

Error 825

This error also falls in the hard I/O error category. When this error is received, it indicates that Windows requested a read and the read failed at least one time. Oftentimes, this is the first indication that there’s trouble brewing in the disk sub-system. This error can also be captured by alerting on errors with a severity of 10.

You’ll find the following error message in the SQL Server error log and Windows event log:

“A read of the file <<FILE NAME>> at offset <<PHYSICAL OFFSET>> succeeded after failing<<FAILURE COUNT>> time(s) with error: <<MESSAGE>>. Additional messages in the SQL Server error log and system event log may provide more detail. This error condition threatens database integrity and must be corrected. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.”

Conclusion

You’re going to receive one, or more, of these errors during your career as a Database Administrator. No one is immune. Bad things happen to hardware, and most of the time there’s not a lot you can do to prevent it. However, you can be prepared to respond.

If you receive one of these errors, don’t wait around before you investigate. You’re response is critical to the health of the data you’re sworn to protect. These types of problems don’t go away on their own. Don’t make the mistake of thinking that they will.

So, keep your storage system firmware and drivers up to date, implement these alerts and systematically plan your response. These actions will give you the best chance of a speedy recovery when database corruption strikes.

Scott Caldwell, SQL Server DBA

• I confirmed the ports were open
• Confirmed DNS
• Even confirmed the netcards

Since we were able to narrow the issue down to bad communication between 2 servers, one of which the DTC and SQL Services had already been restarted, that left us with one and only one potential candidate. So the DTC services were restarted and everything worked.

Checklist

• Confirm that DTC is running properly on servers involved and restart one by one if it’s possible.
• I’d run DTCTest first rather than DTCPing because in my case DTCPing confirmed connectivity. However, I was unable to enlist a transaction which is what I wanted to do at the end of the day.
• If you need additional output, fallback to DTCPing.

Attached are a few more screenshots should you encounter this issue.

Initial Error:

Accompanying Error:

Successful DTCTest display after restarting DTC on the server with the issue:

Thanks,

James Shropshire MCDBA, MCITP SQL Server

However, the profiler has a fatal flaw.  It puts a bit of burden on the system.  Also, you have to run it on the production system in order to get the true test of throughput!  Or you must have a very sophisticated way of playing back the throughput on a test box that matches production.  Hmm.  Then, some DBAs starting thinking…What if I ran the trace from another box!  Viola, the server side trace.  For any production system, the server side trace is a fantastic tool for gaining statistics on deadlocks, indexes, long running queries, and a wealth of other statistical information that can help you improve the setup of your server as well as your code.  Was this the holy grail of tuning that us DBAs had been looking for?

In SQL Server 2005, Microsoft provided the Dynamic Management Views (DMVs).  These provide a constant, real-time view of the database and server.  No trace needs to be run, no data needs to be collected over time, no load on the system, no huge files to figure out where to put on another server, and no rotating file where you are trying to make sure you are not writing over the event that you are trying to capture.  All of the data is right there in the system for you!  You just have query it.  But wait, you say.  I tried to query the views and they weren’t there!   You may need to enable permissions to these views.  You do this through these commands:

• GRANT VIEW SERVER STATE to <Login>
• GRANT VIEW DATABASE STATE to <User>

A few helpful queries I have found to get information from the DMV are listed below. The following shows useful information about all indexes in a database:

SELECT object_id, index_id, user_seeks, user_scans, user_lookups

FROM  sys.dm_db_index_usage_stats

ORDER BY object_id, index_id

The following shows indexes which have never been used:

SELECT object_name(i.object_id),

i.name,

d.user_updates,

d.user_seeks,

d.user_scans,

d.user_lookups

FROM sys.indexes i

LEFT JOIN sys.dm_db_index_usage_stats d

ON d.object_id = i.object_id and

i.index_id = d.index_id and d.database_id = 5

WHERE objectproperty(i.object_id, ‘IsIndexable’) = 1 and

d.index_id is null or

(d.user_updates > 0 and d.user_seeks = 0

And d.user_scans = 0 and d.user_lookups = 0)

ORDER BY object_name(i.object_id)

The following query will tell what each connection is actually performing in SQL Server:

SELECT session_id, status, command, sql_handle, database_id

FROM sys.dm_exec_requests

WHERE session_id >=51

There is a wealth of information available on the DMVs both online and in books.  These views are a DBA’s best friend, and should be the first go-to for tuning opportunities.  There are still times when a server side trace can give more in depth information, such as when performance may differ at different times of day.  But for your average, run-of-the-mill tuning question, nothing can touch the DMVs for their ease of use and their lack of impact to the server.

Katy Park, SQL Server Team Lead

One of the common challenges that all technicians face, no matter what area of IT they work in, is the absolute attention to detail our profession demands. Switch a couple of characters in a script, forget to set your SID, set the wrong flag at the wrong time and the end result usually isn’t very pretty. Many commands we issue on a regular basis are destructive by their very nature.

If you never make mistakes, send me a resume. I’m always looking for a “Patron Saint of IT” here at Remote DBA Experts. It will also save us on travel costs because I’m sure you’ll be able to spread your wings and fly here on your own.

Then there’s the software glitches. The problems that pop up out of the blue and make you go:

“WHAT THE? – How did THAT happen? I’ve done this 317 times in a row and it worked every time.”

For you math majors, here’s my calculation for one of the Foot Principles of IT Support:

CLOSER YOU ARE TO PRODUCTION TURNOVER

+ THE GREATER THE VISIBILITY OF THE PROJECT

= THE MORE LIKELY A PREVIOUSLY UNKNOWN SOFTWARE GLITCH WILL OCCUR

I don’t care what software you are using, you will run into the “only occurs on this release, on this version of the operating system, using this particular feature on the third Tuesday of the sixth month when it’s cloudy outside” BUG. Be sure to expect management to stop by and ask “well, why didn’t you test this on the third Tuesday of the sixth month when it was cloudy outside?”

Finally, there are the hardware problems that we have all come so accustomed to and fond of.   It’s that triple redundant, super fault tolerant, titanium based, twice-the-price component that you just got management to sign off on a few months ago.    The one that your hardware vendor assured you would solve all of your performance and availability problems.   The component they described as “state-of the art and self healing”.  Fixes itself, they said.

The one that forced you to create that 15 page justification document with all the pretty charts and graphs stating that this piece of hardware will still be running in the year 2040.   The component that just decided to flake out and take down your entire online, 10-thousand dollar a minute web-based ordering system.  You then call the vendor and they say “yeah, we’ve heard about that happening occasionally – we thought we got that fixed.”

Root Cause Corrective Action Reports

A customer that is affected by an application outage or slowdown needs to have a firm understating on  what caused the problem, the activities performed to correct the problem and the action items that will be undertaken to mitigate or prevent the problem from occurring again.

The Root Cause Corrective Action Document provides information on the underlying causal factors that generated the problem and a timeline of events that occurred during the problem event. This ensures that all problems are properly analyzed and that all steps are taken to prevent future occurrences.  This is a key component of our problem resolution strategy in addition to obtaining customer feedback on the quality of our problem resolution capabilities.

I can’t stress the importance of using some form of problem notification document.   A customer that is unsure of “what happened” is going to an unhappy customer.   Giving your customer a clear picture of the problem event and the steps you will take to prevent the problem’s reoccurrence  shows them that the quality of their environments is IMPORTANT TO YOU and you do NOT TAKE PROBLEMS LIGHTLY.    The time you spend crafting the Root Cause Correction Action Document will pay big dividends in customer happiness.

The Root Cause Correction Action Document’s components are fairly simple.   Here’s a brief description of each of the sections:

Heading

The heading section contains the customer name, document date, numeric document identifier, the date the problem occurred and the person preparing the document.

Problem Definition

A clear, concise definition of the exact problem.   You need to remember that not everyone reading your document will have a technical background.   Leave the technical mumbo-jumbo out of it.   You are trying to inform your customer of the event NOT confuse them.

Business Impact

What was the impact on their business?  Don’t sugar coat it.   Tell them that the failure caused a 14 hour outage on their production ERP system.   The business impact of the example shown below would be “A 37 minute delay occurred in replication between server ORAPGH and DB2DEL.  During this time DB2DEL reports did not provide current data.”

Event Timelines

This is a chronological timeline of the events that led to the problem (if you know what they are), and the steps that were taken to correct the issue.  Include every step that occurred up to and including verifying that the fixed system was indeed operational.   Here’s an example:

Wednesday May 5, 2010

18:08    Remote DBA Experts’ log monitor for the replication engine determined that replication was not successfully occurring between the two production platforms (ORAPGH and DB2DEL)

18:10     Remote DBA Experts notifies Delaware business units that data replication has stopped and reports being generated will not be current.

18:15     As a recommended action previously provided by the software vendor, Remote DBA Experts stopped and started the replication engine on both platforms.

18:30    Remote DBA Experts verified that the replication engine was running and restarts replication processes.

18:45     Remote DBA Experts verifies that replication was successfully occurring between ORAPGH and DB2DEL.

18:47     Remote DBA Experts begins monitoring the delay to determine the length of time it will take the replication engine to resynchronize the data between ORAPGH and DB2DEL.  Delay estimation is calculated to be 15 minutes.

19:00    Monitors show that both environments are synchronized.

19:05    Remote DBA Experts notifies Delaware business units that replication is occurring and all data is current.

19:10    Logs and trace files are collected and a Severity One problem is initiated with software vendor.

Thursday May 6, 2010

07:00   Software vendor contacts Remote DBA Experts support personnel.  States that problem was caused by a previously unidentified software bug.  Recommends upgrading product to newest release (we’ve never heard that one before).

Problem Root Cause

The underlying causal factor that created the problem event.  In the case above, the root cause was due to a software code issue that caused replication to terminate abnormally.

Contributing Factors

There are times when the problem is exacerbated by contributing factors.  In our example, if a long running job prevented us from successfully stopping the replication engine (leading to a longer outage), we would include a description of that issue in this section.

Resolution

This section contains the actual steps that were taken to correct the problem.   It does not restate the steps in chronological order.   It is a brief description of the activities taken to correct the issue.

Future Prevention

The Future Prevention Section is the most important component of the Root Cause Corrective Action document.  This section provides the steps that you will take as a service provider to ensure that the problem does not reoccur.    It contains a list of action items, the person responsible for completing that action item and a date the action item will be complete.

Signature Section

Signed by the technicians involved with the problem and a member of the service provider management team.

The Importance of Following Up
If you have been reading my previous blogs, you know that we feel so strongly about customer feedback at Remote DBA Experts that we have created a customer feedback strategy called “The Customer Feedback Engine.” We have established multiple communication flows to ensure that we receive feedback from all of the personnel that we support including management, DBAs, developers and end-users.

One of the key strategies is the role our Service Assurance Manager plays.  Remote DBA Experts’ Service Assurance Manager contacts all customers when a problem occurs.  Whether or not we caused the problem is immaterial.  We feel that is our responsibility as a service provider to let our customers know that their application’s performance and availability is important to us.

As my old boss Dan Pizzica used to tell me (when I was a VERY junior technician) “It really doesn’t make a difference who or what broke YOUR database (strong emphasis on the word YOUR). You are the technician who is ultimately responsible for fixing it. The buck stops with you. If you can’t protect your environments, you aren’t doing your job.” We all know he’s absolutely correct.

Wrap-up
It was not the intent of this blog to coerce readers into using the Root Cause Corrective Action information we provide to customers here at Remote DBA Experts verbatim.  It was to promote the benefits that a structured, well thought out problem notification document provides to all of us who are responsible for keeping our customer’s environments highly available and high performance.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

We also know that each organization, group and individual user has their own unique set of value drivers they we’ll use to evaluate the quality of service being provided to them.

It is the responsibility of the service provider to manage their customer’s expectations throughout the entire sales and service delivery life-cycle.   Customers that have a firm understanding of the service offerings, emergency response times, work request lead times and completion times have a much greater chance at being a satisfied customer than those that don’t.

Let’s face it, being a technical service provider isn’t an easy job to begin with.  Your customers will hold you to a higher standard than they do their own in-house personnel.    And they should.  Why complicate the customer management process by not providing your customers with a strong understanding of what they can expect from your organization?

Defining a set of clear, concise Service Level Agreements (SLAs) will help to solidify support requirements and dispel any incorrect assumptions a customer may have on the services your organization provides.  Your customers probably won’t be aware of your current workload and resulting work request lead times until you tell them.

I am a firm believer that every IT unit should meet with their internal and/or external customer base that they support to establish a set of measurable Service Level Agreements.  You don’t have to be a third-party services provider to take advantage of the benefits that a well-defined set of SLAs provides.   I have created a standard SLA document for each company I have been involved with, whether I was an internal employee, consultant or third-party service provider.

Our SLA document at Remote DBA Experts is fairly simple.   It contains Service Categories and individual services.   The SLA entry is clear and concise.   Each entry contains a description of the service, when we will provide the service (upon request, ongoing, reoccurring frequency) and the amount of time the customer can expect the service activity to be completed by.

Service Level Agreement Development:

• Service Identification – The first step is to create a list of services your organization provides to the internal and/or external customer base.   The Information Technology Infrastructure Library, or ITIL as it is most commonly known by, uses the term “Service Catalog” to describe the service listing. ITIL documentation states that IT organizations “must identify what internal and external customers expect” from the service organization before the Service Catalog can be created.   Identifying what your customers want is beyond the scope and intent of this article.   Our focus is to ensure that your customers fully understand the level of service your organization will provide to them.  It is important to list all of the services you offer.  Your SLA document may not contain a minimum service level for all of them, but each one must be evaluated.  You also need to identify if you are guaranteeing any minimum levels of availability and performance for products, applications, hardware components that your unit supports.
• Identify key Service Groupings for the SLA Document – Categorize your list of services into groupings of similar service activities.   At Remote DBA Experts, we have grouped our services into the following basic categories:
  • Customer Integration Activities – Our SLA contains a list of customer integration services that we will perform and their associated target dates.   Not only do we offer customer integration service levels to our customers, we have also defined a set of service levels that we ask all new customers to adhere to during the integration process.   Since we are a remote services provider, we need connectivity, accounts/passwords, environmental descriptions, etc.  Customers understand that we need this information before we can begin servicing their account.   Our SLAs state what we need from the customer and when we need it by.  The remaining service groupings are fairly obvious.
  • Monitoring – Robust monitoring is a key benefit that we provide to our customers here at Remote DBA Experts.  Our SLAs state what we monitor, how often we monitor a particular resource or event, and how quickly we will respond (15 minutes in our case) if a threshold is exceeded or a monitored event occurs.
  • Problem Resolution Activities - One of our SLAs is “Provide expert diagnostic analysis for all database operational problems that arise.”  The frequency is “Upon Problem Identification” and the Service Level is “Troubleshooting begins within 15 minutes of problem identification and continues until problem resolution.”  It tells our customers that when we identify a problem, we will be logged into their systems within 15 minutes and will work on it until it is fixed.   Pretty clear.
  • Database Maintenance and Tuning – This category contains a listing of the various service activities we will perform daily, weekly and monthly to ensure each database is highly available and has optimal performance.
  • Change Management – Our largest category contains all of the changes we can be asked to make to a customer’s database and/or schema.
  • Backup/Recovery and Disaster Recovery – We feel that this set of services is so important to our service offering that we have a service grouping that contains all of the key activities related to making sure our customer’s databases are available when they need them.
  • Installs/Upgrades/Clones – This category contains some of the most time consuming activities that we do for a customer.  The completion times are clearly stated.  We don’t want our customers to enter a ticket to clone their 14 tier Oracle E-Business Environment by the next morning.  Since we service multiple customers here at Remote DBA Experts, clearly stated notification and completion times for time-consuming projects are critical to our ability to service our entire customer base.
  • High Availability Configurations – RAC, Fail Safe, O/S, Data Guard and vendor provided high availability installation services are contained in this category.
  • Customer Management – This category contains all of the activities that relate to informing the customer of our activities.  Entries include meeting frequency, monthly reports, time reports, Root Cause Corrective Action discussions when problems occur and how quickly we will respond to a customer request for additional information.   If it is a communication activity or mechanism with the customer – its in there.

Helpful Hints for Service Level Agreements:

• Make every effort to create a complete list of service activities.  As stated previously, although every activity you identify may not be included in the Service Level Agreement, each one must be evaluated.
• Categorize your list of services into groupings of similar service activities.
• Don’t use high-tech verbiage to describe the service activity. Make the service description clear and concise.  Remember that your audience isn’t as technical as you are.
• During contract negotiations with external customers or meetings with your internal customers, take the time to describe each service entry.   Make sure your customers fully understand each line item.   If your customers find a particular service entry to be confusing or unclear, modify the entry and keep modifying it until it is clear.
• Routine reviews of your service entries and comparing them against changing customer needs are essential to ensure that your services and minimum service levels continue to satisfy customer requirements.
• Don’t create service level metrics for minimum thresholds (uptime and performance levels for example), that are outside of your control.    If you don’t completely control the environment, don’t guarantee a minimum threshold metric for it.   If you must, ensure that the verbiage of the service level metric description clearly states that your organization will only be held responsible for those activities that they do control.  For example, if you are a database unit, you don’t want to miss a minimum service level metric for application availability because of a network problem.  Guarantee only what you can control.

At Remote DBA Experts, we review all of our SLAs on a regular basis.  We also perform in-depth analysis on how well we are accomplishing each of our service level activities defined in our SLA documents.  We use these results to create an internal “report card” that we use to measure our success as a service provider.   If we find that we are weak in some areas, we take steps to correct the deficiency and quickly move on.

There is a direct correlation between how clearly a customer understands your service level guarantees and a successful customer/service provider relationship.  As I stated previously, you probably won’t be meeting all of your customer’s expectations if they (and you) don’t have a firm understanding of what they are.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Verbal and Written Communication Skills
I think people read this blog because they take pride in their work and want to become better at their chosen profession. So here’s my second piece of non-technical advice: The importance of improving your communication skills can not be understated. I don’t care how strong of a technician you are, if you can’t communicate effectively with your peers, you won’t be able to succeed in this profession. In the old days, you might have been able to get by with just your technical skills. That is definitely not the case in today’s business world.

Take a look at your last performance appraisal forms, I’m betting that most of the criteria you are being judged upon depends upon communications. The key words and phrases to look for are “ability to work in a team environment”, “keep supervisors informed”, “maintain good communication with the user community”, “ensure the content of the communication is at the appropriate level for the intended audience”, “provide system and user documentation for projects and system enhancements.” I pulled all of the aforementioned phrases verbatim from one of my own past performance appraisals. I reviewed all of the criteria that I was being evaluated upon and found that almost ninety percent of the items depended upon verbal or written communications.

If you don’t have good communication skills, all is not lost. Like anything else, these skills can be learned. I still consider myself to be only a fair writer. I am in awe of people like Craig Mullins who can just sit down and let the words flow. I often find myself agonizing over every word and sentence. When I first started working in a corporate environment (20 years ago now), my writing skills were terrible. My original career was not database administration, it was construction—a job that didn’t require you to excel at written and verbal communications. One on the job accident, 9 operations and 11 months of vocational rehabilitation training later and I had a new career as a COBOL programmer. I went from working with a construction crew to working with computer programmers. My first employer was a very large and somewhat stuffy financial institution. When I was employed there, men couldn’t leave their floor without wearing their suit coat.   That’s the way it was in the 80′s.

I quickly learned that getting caught wiping your computer screen off with your tie didn’t really show your managers that you were good at thinking “outside of the box.” That was about the only thing I thought that piece of knotted cloth around my neck was good for. For the first six months, I refused to tie them. Being the non-conformist that I was, I just loosened them up, slipped them off and hung them up. Saying that my communication skills were rough around the edges when I started my career would be an understatement. But I had the good fortune of having a manager that understood the importance of both verbal and written communications. I would write a memo, she would correct it with her red pen and send it back to me for a rewrite. Many of them had a “Nice Try!” and a smiley face on top.

After becoming exasperated because of the numerous rewrites (and seeing all of those smiley faces), I thought I had better improve my writing skills. I read books, practiced writing, and became involved with as many company newsletters and related communications as I could. When I asked to join a newsletter, I always started with “I’m not the greatest writer, but I’m trying to learn.” I also asked my peers that worked on the newsletter to critique my work. The more I was critiqued, the better I became.

The same was true with public speaking. My first speech could be described as being “somewhat less than stellar.” Craig Mullins would gently prod me from time to time until he finally convinced me that speaking was something I should be doing from time to time. Craig promised to sit in the back of the room for my first speech and give me hand signals if I was speaking too fast, too slow, too loud or too soft. Halfway through the speech, my knees were knocking and his hands were in constant motion.

I found that like anything else, experience helps. But I will say that my speaking career was not without excitement. I learned that you really shouldn’t drink a carbonated beverage wearing a tie mike that is attached to a set of 6 12-foot speakers. When I was done chugging the pop before the speech, I looked around and saw everyone laughing at the noises I had just made.

I also learned that some podiums are on wheels and those wheels aren’t always locked. I started my first sentence, leaned against the podium, and it began to move. I tripped a little trying to stop the podium from moving and ended up heading for the end of the stage at a very rapid rate. It was a raised stage too, about six feet higher than the first row of seats. As I quickly approached the end of that raised stage, I noticed that the people in the first row were making motions just like the extras did in the old Godzilla movies- right before they got stomped on. Lucky for me one of my work buddies in the first row had the good sense to jump to his feet and stop the podium (and me) from killing a few members of my audience.

At a very large conference, the speaker (who now works for a competitor of mine), that was using the room before my session, left with the tie mike.   The moderators and technicians searched but couldn’t find a spare in time.  What they did find was a 4 foot corded mike that they plugged into the middle of the floor.   I then gave half my speech to 500 participants bent over at the waist.  Luckily, they rounded up the cordless version.   I got over those little snafus and kept plugging away. With each subsequent speech, I started to improve.

The point I am trying to make is that you can improve upon your communication skills. IT shops are no longer evaluating technicians purely on their technical skills. I have seen the soft skill evaluation pendulum swing a little more each year. It is the total package of skills that you bring to the table that you are being evaluated upon. We all know the importance that our technical skill sets have upon our success in this field. But you also need to be well rounded in all of the skill sets your managers are looking for.

In my next set of blogs, we’ll look at some of the different mechanisms we will use to communicate to our customers.  Whether your customer is across the hallway or across the globe, these mechanisms can be used to coordinate your activities with others and keep your customer’s informed of your progress.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

I use change management as a simple example.  Oracle states that 90% of all database failures can be attributed to human error.   Poor communication is certainly one of the problems that we can categorize as a human error.    The last thing you want to hear from an operator when you are requesting the restoration of a tape backup to a different disk is, “I thought you wanted me to restore the files in their original location – not to that spare disk.   I think I just overlaid your prod database…”

Remote DBA Experts provides remote database administration services to organizations across the globe.   We perform 100% of our administrative activities to customers that we rarely see in person.   We have become experts at effective verbal and written communications.   So, let’s talk about communication skills and the important role they play in our careers.

This two-part blog entry will be a somewhat lighthearted look at my own life’s lessons on effective communications (or lack thereof).   In future posts, we’ll look at different mechanisms we can use to communicate and coordinate more effectively with others.  We’ll also learn how we can use effective communications to keep our customer base happy.

It used to be that you could get by with a total lack of interpersonal skills in our profession. Well those days are long gone. If you want to succeed, you’ll need to learn how to communicate effectively and play well with others. These posts contain a few pearls of wisdom, and a story or two, to help you become a well-rounded technician that is on the fast track to a successful career.

The DBA’s Evolving Role
You have read numerous articles on the changing role of the DBA. How the various database vendors are making their products so easy to administer that, sooner or later, DBAs will no longer be required to be expert technicians (or be required at all). If you have been reading this blog, you’ll know that I totally disagree. It is my opinion that database administrators will always need to be just as technically proficient as they had to be in the past. I will agree that database environments are becoming easier to administer. Oracle Grid Control allows us to administer, monitor, tune and troubleshoot an Oracle database without having to go as deep technically as we had to in the past. Although, we won’t have to know as much about the internals as we used to, our skill sets will become much broader in scope.

The database engine is taking on a much more strategic role in most organizations. It provides the mechanism to store physical data along with business rules and executable business logic. The entire application environment (data storage, business rule enforcement, application program storage, communication, system monitoring) is now controlled by the database. Over time, the database engine will store more information related to the understanding of the business, the meaning of the data stored (metadata), and the mechanisms to control and track versions of the database, access programs and related software. As the database’s area of influence expands, so does the DBA’s. Can any one of us predict what IBM, Oracle and Microsoft have up their sleeves in their next “latest and greatest” release? Not me. But THAT is what makes this job exciting. Our area of technical influence will be expanding, not contracting.

The Importance of Soft Skills
Now that I have expressed my opinion on the expanding role of the DBA, let’s talk about the importance of soft skills. Before we begin, I define soft skills as the ability to communicate in both written and verbal forms and the ability to interact with fellow employees in a positive manner. In the past, a technician’s lack of interpersonal skills was often overlooked. The more technically proficient the technician was, the more leeway he or she was given. Let me provide you with a couple of quick examples.

When I first started in this profession, I attended a meeting at a large financial institution that included some pretty high-level representatives from both the business and technical areas. One of the technicians that attended the meeting was a mainframe operating system support technician. The guy was well known to be very good at his job and as nasty a person as you would want to meet. The meeting started and it quickly became apparent that most of the issues being discussed would be about business processes. The O/S expert slammed his pen and pad down on the table and declared, “It looks like this is going to be a waste of my time. I have work to do. Call another meeting with me when you need technical advice.” He then promptly walked out. Being a junior level programmer, I was in awe. I asked my boss after the meeting who he was. He stated “one of our mainframe gurus, don’t act like he does until you get as good as he is.”

A dozen years later, I saw a technician raise his hand at an enterprise-wide IT meeting when the CIO asked (rhetorically I think), who was the most important person in the organization. I think the CIO thought the answer would be “the CEO.” The techie who raised his hand said “I am.” I think every manager attending that meeting shrunk down in his or her seat. When the CIO asked him why, he stated, “When my computers go down, all business stops.” At a previous job, I had a 20-minute conversation with a UNIX admin who never bothered to turn around from her screen to look at me.

Although my examples may be over the top, they show you the mindset that often plagues our profession. As the years have gone by, I have migrated from DBA to DBA Unit Manager and now DBA Operations Manager. I have seen too many excellent technicians end up with a mediocre career because they achieved a reputation for “not playing well with others.” I talk from experience. If you want to excel as a technician, you will need to be technical, but you will also need to work well with others.

Next week, in the conclusion of this two-part blog entry, we’ll continue our discussion on the importance of communication in achieving customer happiness with more words of wisdom and personal experiences.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery