We also know that each organization, group and individual user has their own unique set of value drivers they we’ll use to evaluate the quality of service being provided to them.

It is the responsibility of the service provider to manage their customer’s expectations throughout the entire sales and service delivery life-cycle.   Customers that have a firm understanding of the service offerings, emergency response times, work request lead times and completion times have a much greater chance at being a satisfied customer than those that don’t.

Let’s face it, being a technical service provider isn’t an easy job to begin with.  Your customers will hold you to a higher standard than they do their own in-house personnel.    And they should.  Why complicate the customer management process by not providing your customers with a strong understanding of what they can expect from your organization?

Defining a set of clear, concise Service Level Agreements (SLAs) will help to solidify support requirements and dispel any incorrect assumptions a customer may have on the services your organization provides.  Your customers probably won’t be aware of your current workload and resulting work request lead times until you tell them.

I am a firm believer that every IT unit should meet with their internal and/or external customer base that they support to establish a set of measurable Service Level Agreements.  You don’t have to be a third-party services provider to take advantage of the benefits that a well-defined set of SLAs provides.   I have created a standard SLA document for each company I have been involved with, whether I was an internal employee, consultant or third-party service provider.

Our SLA document at Remote DBA Experts is fairly simple.   It contains Service Categories and individual services.   The SLA entry is clear and concise.   Each entry contains a description of the service, when we will provide the service (upon request, ongoing, reoccurring frequency) and the amount of time the customer can expect the service activity to be completed by.

Service Level Agreement Development:

• Service Identification – The first step is to create a list of services your organization provides to the internal and/or external customer base.   The Information Technology Infrastructure Library, or ITIL as it is most commonly known by, uses the term “Service Catalog” to describe the service listing. ITIL documentation states that IT organizations “must identify what internal and external customers expect” from the service organization before the Service Catalog can be created.   Identifying what your customers want is beyond the scope and intent of this article.   Our focus is to ensure that your customers fully understand the level of service your organization will provide to them.  It is important to list all of the services you offer.  Your SLA document may not contain a minimum service level for all of them, but each one must be evaluated.  You also need to identify if you are guaranteeing any minimum levels of availability and performance for products, applications, hardware components that your unit supports.
• Identify key Service Groupings for the SLA Document – Categorize your list of services into groupings of similar service activities.   At Remote DBA Experts, we have grouped our services into the following basic categories:
  • Customer Integration Activities – Our SLA contains a list of customer integration services that we will perform and their associated target dates.   Not only do we offer customer integration service levels to our customers, we have also defined a set of service levels that we ask all new customers to adhere to during the integration process.   Since we are a remote services provider, we need connectivity, accounts/passwords, environmental descriptions, etc.  Customers understand that we need this information before we can begin servicing their account.   Our SLAs state what we need from the customer and when we need it by.  The remaining service groupings are fairly obvious.
  • Monitoring – Robust monitoring is a key benefit that we provide to our customers here at Remote DBA Experts.  Our SLAs state what we monitor, how often we monitor a particular resource or event, and how quickly we will respond (15 minutes in our case) if a threshold is exceeded or a monitored event occurs.
  • Problem Resolution Activities - One of our SLAs is “Provide expert diagnostic analysis for all database operational problems that arise.”  The frequency is “Upon Problem Identification” and the Service Level is “Troubleshooting begins within 15 minutes of problem identification and continues until problem resolution.”  It tells our customers that when we identify a problem, we will be logged into their systems within 15 minutes and will work on it until it is fixed.   Pretty clear.
  • Database Maintenance and Tuning – This category contains a listing of the various service activities we will perform daily, weekly and monthly to ensure each database is highly available and has optimal performance.
  • Change Management – Our largest category contains all of the changes we can be asked to make to a customer’s database and/or schema.
  • Backup/Recovery and Disaster Recovery – We feel that this set of services is so important to our service offering that we have a service grouping that contains all of the key activities related to making sure our customer’s databases are available when they need them.
  • Installs/Upgrades/Clones – This category contains some of the most time consuming activities that we do for a customer.  The completion times are clearly stated.  We don’t want our customers to enter a ticket to clone their 14 tier Oracle E-Business Environment by the next morning.  Since we service multiple customers here at Remote DBA Experts, clearly stated notification and completion times for time-consuming projects are critical to our ability to service our entire customer base.
  • High Availability Configurations – RAC, Fail Safe, O/S, Data Guard and vendor provided high availability installation services are contained in this category.
  • Customer Management – This category contains all of the activities that relate to informing the customer of our activities.  Entries include meeting frequency, monthly reports, time reports, Root Cause Corrective Action discussions when problems occur and how quickly we will respond to a customer request for additional information.   If it is a communication activity or mechanism with the customer – its in there.

Helpful Hints for Service Level Agreements:

• Make every effort to create a complete list of service activities.  As stated previously, although every activity you identify may not be included in the Service Level Agreement, each one must be evaluated.
• Categorize your list of services into groupings of similar service activities.
• Don’t use high-tech verbiage to describe the service activity. Make the service description clear and concise.  Remember that your audience isn’t as technical as you are.
• During contract negotiations with external customers or meetings with your internal customers, take the time to describe each service entry.   Make sure your customers fully understand each line item.   If your customers find a particular service entry to be confusing or unclear, modify the entry and keep modifying it until it is clear.
• Routine reviews of your service entries and comparing them against changing customer needs are essential to ensure that your services and minimum service levels continue to satisfy customer requirements.
• Don’t create service level metrics for minimum thresholds (uptime and performance levels for example), that are outside of your control.    If you don’t completely control the environment, don’t guarantee a minimum threshold metric for it.   If you must, ensure that the verbiage of the service level metric description clearly states that your organization will only be held responsible for those activities that they do control.  For example, if you are a database unit, you don’t want to miss a minimum service level metric for application availability because of a network problem.  Guarantee only what you can control.

At Remote DBA Experts, we review all of our SLAs on a regular basis.  We also perform in-depth analysis on how well we are accomplishing each of our service level activities defined in our SLA documents.  We use these results to create an internal “report card” that we use to measure our success as a service provider.   If we find that we are weak in some areas, we take steps to correct the deficiency and quickly move on.

There is a direct correlation between how clearly a customer understands your service level guarantees and a successful customer/service provider relationship.  As I stated previously, you probably won’t be meeting all of your customer’s expectations if they (and you) don’t have a firm understanding of what they are.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

I started my career working in a mainframe environment. Fellow blogger Craig Mullins and I administered DB2 databases on huge, big iron platforms. Platforms that supported thousands (upon thousands) of concurrent users. One of the benefits of having a background in mainframe support is that it taught us the importance of change management best practices.

Craig and I learned that a key ingredient of a trouble-free mainframe environment is ensuring that there are no “surprises” when a change is implemented in production. Changes that can affect the day-to-day business operations of an entire organization. Throughout my career, I have applied these “mainframe style” best practices to all other database ecosystems that I was responsible for supporting.

It works. The first company I applied these best practices to was selected by Oracle as a “Showcase Environment”. This was back in the old days, when Scott’s tiger was just a cub. Oracle identified shops that it thought had rock-solid support environments and asked them to host visitors from other organizations that wanted to create their own high-quality support infrastructures.

We apply these same principles to our remote service offerings here at Remote DBA Experts. We support hundreds of developers around-the-clock. If a customer does not have a strong set of application development change management procedures in place, we will help them create one during our transition activities.

I thought I would provide you with a few helpful hints and tips on change management best practices.

Database Design Reviews
One of my earlier blogs provides information on database design reviews. Database design review meetings foster effective communications between the DBA unit, system support personnel and application developers throughout the entire application design and implementation process. When Oracle design issues are addressed early in the development lifecycle, problems are minimized and the migration from test to production is more easily accomplished.

If you haven’t read the design review blog, you should. I’m intentionally not covering the importance of rigorous testing of any change before it is implemented in production because it is covered in-depth in the design review blog. From simple changes to new application implementations, there is simply no reason not to perform the test, review, change, test, review, change iteration lifecycle. Although the blog covers new application implementations, the blog will show you how important I think it is to follow a rigorous test plan.

Proceduralize the Change Request Process
Database administrators usually support different business units with each unit having their own set of unique procedural requirements. Formalizing and documenting the change request process minimizes the potential for miscommunication between the business units, application development areas and the database administration unit. This is even more important to remote database services providers.

The notification lead-time required for the database administration team to perform a requested change should be documented and distributed to business and application development units. This will prevent your team for getting a request to migrate a database from test to production in the morning with a required date for that afternoon. Of course, we all know that never happens.

If your organization doesn’t have a formal change request process in place, (and many shops don’t) create your own! There are dozens of change management and source code versioning tools available on the market today. The prices can range from thousands to tens of thousands of dollars.

Although I highly recommend these types of products, I wouldn’t let the lack of having one prevent me from formalizing the change management process. You do the best with what you have.

OK, so you don’t have the good fortune of having a formal change management process in place. What do you do?

Althouth our internal practices here at Remote DBA Experts differ because of the advanced communication and ticketing products we offer to our customers, I’ve come up with a set of “time tested” best practices and procedures that can be used by any shop. Since I have over 25 years of experience with Oracle, that’s a lot of time to test!

You can begin the formalization of the change request process by:

• Creating standardized change request documents
• Establishing change management meetings
• Creating Service Level Agreements (SLAs), which include change request turnaround times

Standardized Change Request Documents
Standardized request documents help to increase the quality of the change request process. The forms are sent to the database administration unit by the application owner of the data to be processed. Any requests not sent or signed off by the application owner should be rejected. Keep copies of all completed work requests for auditing purposes. Application owners can be virtually any member of the organization that is identified as having the authority to sign off on change requests. The most common persons are application development team leaders, section heads, department heads, etc..

Each request form contains the following common information:

• Form identifier – naming convention that allows the form to be easily identified
• Application name
• Database name
• Name and contact information of the person requesting the change
• Request date
• Required date
• Application owner signoff
• Data security signoff (if required by shop standards)
• A free form request area for nonstandard requests
• An area that the DBA will fill out when executing the change that contains the following information:
  • DBA executing change
  • DBA contact information
  • Date and time change was processed
  • Verification procedures followed (very important)

Here are a few examples of specific forms that will help formalize the change request process:

Oracle Authorization Request Form
This form is used for requesting authorization changes to the Oracle database environment. Here’s an example of one of the earliest Oracle authorization forms I created over 20 years ago. This would still work!

The Oracle Authorization Request Form contains the following information pertinent to Oracle authorization requests:

• Grantee listing for security grant or revoke
• Type of security granted or revoked

Oracle Test Environment Change Request Form
The Oracle Test Environment Change Request Form will be used for requesting physical changes to Oracle test databases. It is used for requesting both new objects (tables, indexes, views, stored programs, etc.) and object alterations (adding new columns, changing datatypes, adding columns to indexes, etc.). In addition, the form notifies the DBA team to perform ad-hoc activities (test database refreshes, exports, etc..) in test environments.

The Oracle Test Environment Change Request Form contains the following information pertinent to Oracle change requests:

• Schema owner of object to be changed or created
• Object name to be changed or created
• Object type (i.e. table, index, view) of the object to be changed or created
• Detailed description of change requested
• Data administration sign off for new data objects
• Free from area for ad-hoc activities

Oracle Production Environment Change Request Form
This form will be used for requesting the migration of Oracle objects (databases, tablespaces, tables, indexes, etc.) from test to production and the alteration of existing production objects. For new database implementations, please see my blog titled “Database Design Reviews”. In addition, the form notifies the DBA team to perform ad-hoc activities in production environments.

Each Production Environment Change Request Form must have an associated Oracle Test Environment Change Request Form counterpart. If the change wasn’t made in test, you don’t implement it in production. To facilitate this process, the identifier for the test change request form that was used to notify the DBA team to create or alter the test objects should be provided on the production change request form.

The Production Environment Change Request Form contains the following information pertinent to production environments:

• Oracle Test Database Change Request Form. Allows DBA to determine if the change was implemented in test. If no change request is found, the DBA needs to determine the reason why
• Object name of the object to be migrated or changed
• Object type for the object to be migrated or changed
• What time these activities should be performed
• The name of the database the object will be migrated from (source database)
• The name of the database the object creation/alteration will be implemented in (target database)
• A freeform area for special processing required during the migration or alteration process
• Free from area for ad-hoc activities

With all of the advanced workflow, ticketing and communication products on the market today, there are numerous options available to the DBA wanting to proceduralize the notification process. Whatever product you choose, its the information that you capture and the change management workflow process you create that is important.

Change Management Meetings
If you read my earlier blog on database design review meetings, you know I’m a proponent of constant communication between all units that are involved in the change management process. Here at Remote Database Experts we make sure we attend as many of our customers change management meetings as we can. If its a database change, we want to be there to discuss it.

How often should you hold these change management meetings? As often as you implement objects in production. If your organization makes changes to production environments on a daily basis, the meetings should be held daily. This is not as big of an imposition on your time as you may think. We provide remote database services for several very large organizations that have these change management meetings on a daily basis. The process takes about 15 to 20 minutes. Not a lot of time spent to ensure that everyone knows what is happening.

To shorten the amount of time these meetings consume and to make them as productive as possible, the following discussion items should be a standard part of the meeting’s agenda:

• Application name being changed
• Date and time change will be implemented
• Change description
• Potential business impact if the changes don’t go as expected (include both units affected and how they will be affected)
• Backoff procedures
• Requestor
• Tested by

Service Level Agreements
Identifying support needs and expectations is required to provide high quality support. You probably won’t be meeting all of your customer’s expectations if you don’t know what any of them are. As stated previously, each application has it own unique set of support requirements and expectations. Service Level Agreements (SLA) help to solidify support requirements and dispel any inflated expectations a business or application development unit may have. They probably won’t be aware of your current workload and resulting work request lead times until you tell them. The DBA team lead should meet with each unit supported to establish a set of measurable Service Level Agreements that include work request lead times, support activities required and application performance and availability objectives.

Wrapup
This is by no means an all inclusive list of activities you need to perform to standardize the change request process. It is intended to give you a head start in the right direction.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery