You’ll find references regarding these in blogs across the Internet and some are more useful than others. Obviously, these are unsupported commands and should only be used with great care.

TRACE FLAGS

There are quite a few undocumented trace flags in SQL Server.  However, I’m only going to touch on four today. They are trace flags 3004, 3014 3604 and 3605.

These can be enabled for the current session with the following commands:

– Enable the trace flags

DBCC TRACEON(3004);

DBCC TRACEON(3014);

DBCC TRACEON(3604);

DBCC TRACEON(3605);

GO

These trace flags are going to be used in conjunction with a database restore so we can get an idea about what SQL Server is doing during the process. We’ll be looking at output in the message output screen and output sent to the SQL Server error log.

Before we go much further, I want to briefly explain the purpose of these flags and why it’s important to use them together to get the information we’re looking for.

Trace Flag 3004

Most Database Administrators are aware of instant file initialization. In a nutshell, when instant file initialization is enabled the data files do not need to be zeroed out during creation. This can save an incredible amount of time during the restoration of VLDBs. As you can imagine, the zeroing out of a 1 TB data file can take a very long time.

Trace flag 3004 turns on information regarding instant file initialization. Enabling this trace flag will not make this information available to view. You will still need to turn on trace flag 3605 to send this information to the error log.

Trace Flag 3014

Trace flag 3014 provides detailed information regarding the steps performed during the backup and restore process. Normally, SQL Server only provides a limited amount of information in the error log regarding these processes. By enabling this trace flag you’ll be able to see some very detailed and interesting information.

Trace Flag 3604

Trace flag 3604 can be used under a variety of circumstances. If you’ve ever used DBCC IND or DBCC PAGE then you’ve probably already used trace flag 3604. It simply informs SQL Server to send some DBCC output information to the screen instead of the error log. In many cases, you have to use this trace flag to see any output at all.

Trace Flag 3605

Trace flag 3605 will send some DBCC output to the error log. This trace flag needs to be enabled to see the instant file initialization information made available by trace flag 3004.

Restore without Trace Flags

I created a full backup of the AdventureWorks database and then restored it with the following command:

– Restore the database

RESTORE DATABASE AdventureWorks

FROM DISK = ‘C:\TEMP\ADVENTUREWORKS.BAK’

GO

Without these trace flags the following information is returned to the message output screen:

You can also see very similar information in the error log. I returned the contents of the error log with the following command:

– Read the error log

EXEC xp_readerrorlog;

GO

Restore with Trace Flags

I’m going to perform the restore again. This time my four trace flags have been enabled and I’m hoping to see some additional information in both the message output window and the error log.

In order to make sure the error log is easy to read, I’ve cycled it with the following command:

– Cycle the error log

EXEC sp_cycle_errorlog;

GO

The next step is to execute the restore again.

– Restore the database

RESTORE DATABASE AdventureWorks

FROM DISK = ‘C:\TEMP\ADVENTUREWORKS.BAK’

GO

After the restore is finished, and with the log cleared and the trace flags enabled, we get the following information in the message output window:

It’s easy to see that this output is far more detailed when compared to what we received during our last restore. You notice that there are quite a few additional steps that were not reported when the trace flags were not enabled.

But wait, we also need to check the error log to see what additional information is available there too.

– Read the error log

EXEC xp_readerrorlog;

GO

The most interesting information in the error log is in regard to instant file initialization. You can see that SQL Server is zeroing out the data file during the container prepare process. This means we’re not benefiting from instant file initialization.

Conclusion

It’s important to understand that SQL Server purposely hides this level of detail from us on a daily basis. Arguably, this information isn’t really that necessary. However, if you feel as though you need to have a greater understanding about what’s going on inside the database engine then this is a good starting point on your journey.

Scott Caldwell, SQL Server DBA

The Usual Suspect

Generally speaking, the majority of database corruption is caused by hardware errors in the disk sub-system. Our database files are sitting on our SAN, or DAS, and are happily going about their lives without any problem until something goes awry. You may not even know there is a problem until it’s too late and a database is marked suspect.

The most important thing to can do, outside of monitoring for these problems, is to keep your storage system’s firmware and drivers up to date and consistent within the system. Sign-up for notifications with your vendor so you will receive these notices, and when you received them, read the little text file that explains what has changed and what will happen if you don’t update. I can’t tell you how often I’ve read through these and run across verbiage like, “previous version can cause data loss.”

These systems aren’t perfect. Stay on top of your firmware and driver updates or you may be turned down when you make that middle of the night support call and need their help.

Automate the Eights

I’ve got a bag of alerts that I like to implement on each SQL Server that I administer. Three of these alerts are for errors 823, 824 & 825.

Error 823

Sometimes referred to as a hard I/O error, this is alerting SQL Server, and you, that a read or write error has occurred. In the case of a read error, Windows has already attempted a retry four times before sending the message. This error can also be captured by alerting on errors with a severity of 24.

This error will be accompanied by the following entry in the SQL Server error log and Windows event log:

“The operating system returned error 823 to SQL Server during a <<MESSAGE>> at offset <<PHYSICAL OFFSET>> in file <<FILE NAME>>. Additional messages in the SQL Server error log and system event log may provide more detail. This is a severe system-level error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online”.

Error 824

Sometimes referred to as a logical consistency error, or soft I/O error, this indicates that Windows was able to read the page from disk but SQL Server has discovered that the page has a problem. This error can be caused when SQL Server determines that there was a checksum error or a torn bits error when it read the page from disk. This is dependent on the page verification option your database is using. This error can also be captured by alerting on errors with a severity of 24.

No matter which occurred, there’s a problem with the page and the following error message will be reported in the SQL Server error log and the Windows event log:

“SQL Server detected a logical consistency-based I/O error: <<ERROR TYPE DESCRIPTION>>. It occurred during a <<Read/Write>> of page <<PAGEID>> in database ID <<DBID>> at offset <<PHYSICAL OFFSET>> in file <<FILE NAME>>. Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.”

Error 825

This error also falls in the hard I/O error category. When this error is received, it indicates that Windows requested a read and the read failed at least one time. Oftentimes, this is the first indication that there’s trouble brewing in the disk sub-system. This error can also be captured by alerting on errors with a severity of 10.

You’ll find the following error message in the SQL Server error log and Windows event log:

“A read of the file <<FILE NAME>> at offset <<PHYSICAL OFFSET>> succeeded after failing<<FAILURE COUNT>> time(s) with error: <<MESSAGE>>. Additional messages in the SQL Server error log and system event log may provide more detail. This error condition threatens database integrity and must be corrected. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.”

Conclusion

You’re going to receive one, or more, of these errors during your career as a Database Administrator. No one is immune. Bad things happen to hardware, and most of the time there’s not a lot you can do to prevent it. However, you can be prepared to respond.

If you receive one of these errors, don’t wait around before you investigate. You’re response is critical to the health of the data you’re sworn to protect. These types of problems don’t go away on their own. Don’t make the mistake of thinking that they will.

So, keep your storage system firmware and drivers up to date, implement these alerts and systematically plan your response. These actions will give you the best chance of a speedy recovery when database corruption strikes.

Scott Caldwell, SQL Server DBA

However, the profiler has a fatal flaw.  It puts a bit of burden on the system.  Also, you have to run it on the production system in order to get the true test of throughput!  Or you must have a very sophisticated way of playing back the throughput on a test box that matches production.  Hmm.  Then, some DBAs starting thinking…What if I ran the trace from another box!  Viola, the server side trace.  For any production system, the server side trace is a fantastic tool for gaining statistics on deadlocks, indexes, long running queries, and a wealth of other statistical information that can help you improve the setup of your server as well as your code.  Was this the holy grail of tuning that us DBAs had been looking for?

In SQL Server 2005, Microsoft provided the Dynamic Management Views (DMVs).  These provide a constant, real-time view of the database and server.  No trace needs to be run, no data needs to be collected over time, no load on the system, no huge files to figure out where to put on another server, and no rotating file where you are trying to make sure you are not writing over the event that you are trying to capture.  All of the data is right there in the system for you!  You just have query it.  But wait, you say.  I tried to query the views and they weren’t there!   You may need to enable permissions to these views.  You do this through these commands:

• GRANT VIEW SERVER STATE to <Login>
• GRANT VIEW DATABASE STATE to <User>

A few helpful queries I have found to get information from the DMV are listed below. The following shows useful information about all indexes in a database:

SELECT object_id, index_id, user_seeks, user_scans, user_lookups

FROM  sys.dm_db_index_usage_stats

ORDER BY object_id, index_id

The following shows indexes which have never been used:

SELECT object_name(i.object_id),

i.name,

d.user_updates,

d.user_seeks,

d.user_scans,

d.user_lookups

FROM sys.indexes i

LEFT JOIN sys.dm_db_index_usage_stats d

ON d.object_id = i.object_id and

i.index_id = d.index_id and d.database_id = 5

WHERE objectproperty(i.object_id, ‘IsIndexable’) = 1 and

d.index_id is null or

(d.user_updates > 0 and d.user_seeks = 0

And d.user_scans = 0 and d.user_lookups = 0)

ORDER BY object_name(i.object_id)

The following query will tell what each connection is actually performing in SQL Server:

SELECT session_id, status, command, sql_handle, database_id

FROM sys.dm_exec_requests

WHERE session_id >=51

There is a wealth of information available on the DMVs both online and in books.  These views are a DBA’s best friend, and should be the first go-to for tuning opportunities.  There are still times when a server side trace can give more in depth information, such as when performance may differ at different times of day.  But for your average, run-of-the-mill tuning question, nothing can touch the DMVs for their ease of use and their lack of impact to the server.

Katy Park, SQL Server Team Lead

Verbal and Written Communication Skills
I think people read this blog because they take pride in their work and want to become better at their chosen profession. So here’s my second piece of non-technical advice: The importance of improving your communication skills can not be understated. I don’t care how strong of a technician you are, if you can’t communicate effectively with your peers, you won’t be able to succeed in this profession. In the old days, you might have been able to get by with just your technical skills. That is definitely not the case in today’s business world.

Take a look at your last performance appraisal forms, I’m betting that most of the criteria you are being judged upon depends upon communications. The key words and phrases to look for are “ability to work in a team environment”, “keep supervisors informed”, “maintain good communication with the user community”, “ensure the content of the communication is at the appropriate level for the intended audience”, “provide system and user documentation for projects and system enhancements.” I pulled all of the aforementioned phrases verbatim from one of my own past performance appraisals. I reviewed all of the criteria that I was being evaluated upon and found that almost ninety percent of the items depended upon verbal or written communications.

If you don’t have good communication skills, all is not lost. Like anything else, these skills can be learned. I still consider myself to be only a fair writer. I am in awe of people like Craig Mullins who can just sit down and let the words flow. I often find myself agonizing over every word and sentence. When I first started working in a corporate environment (20 years ago now), my writing skills were terrible. My original career was not database administration, it was construction—a job that didn’t require you to excel at written and verbal communications. One on the job accident, 9 operations and 11 months of vocational rehabilitation training later and I had a new career as a COBOL programmer. I went from working with a construction crew to working with computer programmers. My first employer was a very large and somewhat stuffy financial institution. When I was employed there, men couldn’t leave their floor without wearing their suit coat.   That’s the way it was in the 80′s.

I quickly learned that getting caught wiping your computer screen off with your tie didn’t really show your managers that you were good at thinking “outside of the box.” That was about the only thing I thought that piece of knotted cloth around my neck was good for. For the first six months, I refused to tie them. Being the non-conformist that I was, I just loosened them up, slipped them off and hung them up. Saying that my communication skills were rough around the edges when I started my career would be an understatement. But I had the good fortune of having a manager that understood the importance of both verbal and written communications. I would write a memo, she would correct it with her red pen and send it back to me for a rewrite. Many of them had a “Nice Try!” and a smiley face on top.

After becoming exasperated because of the numerous rewrites (and seeing all of those smiley faces), I thought I had better improve my writing skills. I read books, practiced writing, and became involved with as many company newsletters and related communications as I could. When I asked to join a newsletter, I always started with “I’m not the greatest writer, but I’m trying to learn.” I also asked my peers that worked on the newsletter to critique my work. The more I was critiqued, the better I became.

The same was true with public speaking. My first speech could be described as being “somewhat less than stellar.” Craig Mullins would gently prod me from time to time until he finally convinced me that speaking was something I should be doing from time to time. Craig promised to sit in the back of the room for my first speech and give me hand signals if I was speaking too fast, too slow, too loud or too soft. Halfway through the speech, my knees were knocking and his hands were in constant motion.

I found that like anything else, experience helps. But I will say that my speaking career was not without excitement. I learned that you really shouldn’t drink a carbonated beverage wearing a tie mike that is attached to a set of 6 12-foot speakers. When I was done chugging the pop before the speech, I looked around and saw everyone laughing at the noises I had just made.

I also learned that some podiums are on wheels and those wheels aren’t always locked. I started my first sentence, leaned against the podium, and it began to move. I tripped a little trying to stop the podium from moving and ended up heading for the end of the stage at a very rapid rate. It was a raised stage too, about six feet higher than the first row of seats. As I quickly approached the end of that raised stage, I noticed that the people in the first row were making motions just like the extras did in the old Godzilla movies- right before they got stomped on. Lucky for me one of my work buddies in the first row had the good sense to jump to his feet and stop the podium (and me) from killing a few members of my audience.

At a very large conference, the speaker (who now works for a competitor of mine), that was using the room before my session, left with the tie mike.   The moderators and technicians searched but couldn’t find a spare in time.  What they did find was a 4 foot corded mike that they plugged into the middle of the floor.   I then gave half my speech to 500 participants bent over at the waist.  Luckily, they rounded up the cordless version.   I got over those little snafus and kept plugging away. With each subsequent speech, I started to improve.

The point I am trying to make is that you can improve upon your communication skills. IT shops are no longer evaluating technicians purely on their technical skills. I have seen the soft skill evaluation pendulum swing a little more each year. It is the total package of skills that you bring to the table that you are being evaluated upon. We all know the importance that our technical skill sets have upon our success in this field. But you also need to be well rounded in all of the skill sets your managers are looking for.

In my next set of blogs, we’ll look at some of the different mechanisms we will use to communicate to our customers.  Whether your customer is across the hallway or across the globe, these mechanisms can be used to coordinate your activities with others and keep your customer’s informed of your progress.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

I use change management as a simple example.  Oracle states that 90% of all database failures can be attributed to human error.   Poor communication is certainly one of the problems that we can categorize as a human error.    The last thing you want to hear from an operator when you are requesting the restoration of a tape backup to a different disk is, “I thought you wanted me to restore the files in their original location – not to that spare disk.   I think I just overlaid your prod database…”

Remote DBA Experts provides remote database administration services to organizations across the globe.   We perform 100% of our administrative activities to customers that we rarely see in person.   We have become experts at effective verbal and written communications.   So, let’s talk about communication skills and the important role they play in our careers.

This two-part blog entry will be a somewhat lighthearted look at my own life’s lessons on effective communications (or lack thereof).   In future posts, we’ll look at different mechanisms we can use to communicate and coordinate more effectively with others.  We’ll also learn how we can use effective communications to keep our customer base happy.

It used to be that you could get by with a total lack of interpersonal skills in our profession. Well those days are long gone. If you want to succeed, you’ll need to learn how to communicate effectively and play well with others. These posts contain a few pearls of wisdom, and a story or two, to help you become a well-rounded technician that is on the fast track to a successful career.

The DBA’s Evolving Role
You have read numerous articles on the changing role of the DBA. How the various database vendors are making their products so easy to administer that, sooner or later, DBAs will no longer be required to be expert technicians (or be required at all). If you have been reading this blog, you’ll know that I totally disagree. It is my opinion that database administrators will always need to be just as technically proficient as they had to be in the past. I will agree that database environments are becoming easier to administer. Oracle Grid Control allows us to administer, monitor, tune and troubleshoot an Oracle database without having to go as deep technically as we had to in the past. Although, we won’t have to know as much about the internals as we used to, our skill sets will become much broader in scope.

The database engine is taking on a much more strategic role in most organizations. It provides the mechanism to store physical data along with business rules and executable business logic. The entire application environment (data storage, business rule enforcement, application program storage, communication, system monitoring) is now controlled by the database. Over time, the database engine will store more information related to the understanding of the business, the meaning of the data stored (metadata), and the mechanisms to control and track versions of the database, access programs and related software. As the database’s area of influence expands, so does the DBA’s. Can any one of us predict what IBM, Oracle and Microsoft have up their sleeves in their next “latest and greatest” release? Not me. But THAT is what makes this job exciting. Our area of technical influence will be expanding, not contracting.

The Importance of Soft Skills
Now that I have expressed my opinion on the expanding role of the DBA, let’s talk about the importance of soft skills. Before we begin, I define soft skills as the ability to communicate in both written and verbal forms and the ability to interact with fellow employees in a positive manner. In the past, a technician’s lack of interpersonal skills was often overlooked. The more technically proficient the technician was, the more leeway he or she was given. Let me provide you with a couple of quick examples.

When I first started in this profession, I attended a meeting at a large financial institution that included some pretty high-level representatives from both the business and technical areas. One of the technicians that attended the meeting was a mainframe operating system support technician. The guy was well known to be very good at his job and as nasty a person as you would want to meet. The meeting started and it quickly became apparent that most of the issues being discussed would be about business processes. The O/S expert slammed his pen and pad down on the table and declared, “It looks like this is going to be a waste of my time. I have work to do. Call another meeting with me when you need technical advice.” He then promptly walked out. Being a junior level programmer, I was in awe. I asked my boss after the meeting who he was. He stated “one of our mainframe gurus, don’t act like he does until you get as good as he is.”

A dozen years later, I saw a technician raise his hand at an enterprise-wide IT meeting when the CIO asked (rhetorically I think), who was the most important person in the organization. I think the CIO thought the answer would be “the CEO.” The techie who raised his hand said “I am.” I think every manager attending that meeting shrunk down in his or her seat. When the CIO asked him why, he stated, “When my computers go down, all business stops.” At a previous job, I had a 20-minute conversation with a UNIX admin who never bothered to turn around from her screen to look at me.

Although my examples may be over the top, they show you the mindset that often plagues our profession. As the years have gone by, I have migrated from DBA to DBA Unit Manager and now DBA Operations Manager. I have seen too many excellent technicians end up with a mediocre career because they achieved a reputation for “not playing well with others.” I talk from experience. If you want to excel as a technician, you will need to be technical, but you will also need to work well with others.

Next week, in the conclusion of this two-part blog entry, we’ll continue our discussion on the importance of communication in achieving customer happiness with more words of wisdom and personal experiences.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Since we are a remote services provider, our customers’ businesses range the spectrum from heavy industry to high technology.   We understand that each of our customers has a unique set of value drivers upon which they evaluate the quality of services we provide to them.  We are also aware that our customer’s service delivery expectations are constantly affected by changing business and economic drivers.  The types of services we provide must be tuned and tweaked continuously to ensure that we meet these fluid service delivery expectations.

Once you have collected your own set of customer evaluations, you need to review the results and tally them to identify any common issues that are occurring. Epi Torres, my Remote DBA Experts’ co-blogger, has already identified the key evaluation criteria that are used to evaluate services providers: Timeliness, Efficiency, Effectiveness, Responsiveness, Quality and Integrity. You can categorize your own customer responses according to the key evaluation criteria we have identified or create your own. Once you create the categories, your next step is to perform a thorough analysis of the responses in each of the them.  If you identified a common theme of recommendations, your next step is to rank them in order of importance to your customers.

As you evaluate the categorized responses, you will need to take external influences into consideration that may be affecting your ability to provide high-quality support (inadequate staffing levels, time consuming projects, technical limitations of the environment). One of the key benefits of these documented responses is that this information can be passed up the management chain for further analysis and recommendations. For example, if you have too few DBAs and too many work requests, this additional documentation will notify your management team that the quality of work is suffering.  If additions to staff are hard to come by due to economic conditions, you have at least made both your management team and customer base the importance of setting realistic deadlines and prioritizing workloads.

Action plans to improve in weak areas are created during the evaluation process. Once agreed upon, the internal procedural and process changes can be implemented to tune and tweak the service delivery mechanisms.  A formal document that describes the issues identified and provides details on the changes that will take place to address them can then be distributed to the customers involved in the evaluation and the DBA team’s management chain.

It is important to also address specific customer recommendations that were contained in the customer responses.  Even though a specific issue was not identified as a common theme that exists across the entire customer base, they must still be addressed to ensure that individual customer’s happiness.  Is the customer’s expectation realistic within the confines of the external influences?  If it is, the recommendation needs to be addressed.

Follow-up meetings, designed to ensure that the service delivery changes are having the desired outcome, should be scheduled a few weeks after the action items document is distributed. The meetings should be held with the customers that participated in the original review process.  The discussion will address recommendations that were identified to be common amongst the entire customer base as well as the individual recommendations identified by that particular customer.

The process then becomes iterative in its nature. We have developed our own cyclical customer feedback process here at Remote DBA Experts.  We feel so strongly about this iterative approach to customer feedback that we have created a standardized customer feedback strategy called “The Customer Feedback Engine.” We have established multiple communication flows to ensure that we receive feedback from all of the folks that we support including management, DBAs, developers and end-users. We continuously approach our customers to gather feedback to improve our services.   We also look for new, innovative ways to gather customer feedback, process it, improve our approach to service delivery and then communicate those improvements to our customers.

You need to know what your customers are expecting from you.  You also need to know how they feel about the quality of support that you are currently providing. Without that information you will never be sure if you are providing the highest level of quality support possible.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Since we support many different organizations in virtually every market silo (high technology, heavy manufacturing, retail, health care, etc.), it is obvious to us that each of our customers has a unique set of value drivers that they use to evaluate the quality of our service.   If we don’t understand what they are, how can we be sure that we are meeting their needs? The answer is that we can’t.

Since I work for a remote services provider, we live and die by customer feedback.   But do DBAs that work for a single business organization need their own “Customer Feedback Engine”?   The answer is ABSOLUTELY.

Each group that you support has their own set of value drivers.  You have to understand what they want.  If you don’t, you will be viewed as just a mere technician and that is exactly where you’ll stay throughout your career.  This is NOT an optimal career path. Database administrators have a highly visible role in every organization. You can take advantage of that role to be viewed as someone who is a key player, as opposed to a mere technician.

You can obtain customer feedback by any number of physical mechanisms but it all boils down to this.  You need to put your ego aside, understand that you need to obtain constructive criticism to get better at what you get paid to do and ask your customers. Some of the more popular choices are DBA “report cards” or surveys that are sent to your customer base.   I have used report cards with much success in the past.

A helpful hint with report cards is to make sure you include a field for the participant to include their business unit but not require a signature.  This will help the participants feel more comfortable when they fill out the survey and provide you with a higher quality source of feedback information that you can use to tune and tweak your service delivery methods.

Since we support so many different customers, let me give you some examples of the questions that you need to ask:

• General Support - The intent of the first question is to obtain a general, high-level understanding of how you are doing.  It opens up the conversation to allow you to obtain feedback that may not be obtained from the more specific questions you will ask.
• Responsiveness - Are you responding quickly enough to their requests?  Do you complete tasks when they are needed?
• Communication - There are so many aspects to this subject, it is important to provide the participants with specific examples.
  • Frequency - Do you provide your customers with information on a timely basis?  Are you keeping them aware of your accomplishments with clear updates on large projects?  Do you inform them when you complete daily work requests in a timely manner?  Are you providing your customers with the appropriate status reports on long-running problems that are affecting their application?
  • Content - Does the level of communication you are providing to your customer match their technical background?  Does the language you use seem too technical, not technical enough, too high level or are you delving too deep into the details?
  • Clarity - When you are providing or asking for information, do they understand what you are telling them or what you want from them?
  • Communication Mechanisms- What communication transfer methods do they feel comfortable with?  Do they prefer e-mails, ticket updates, phone calls or quick face-to-face meeting?   Our customers range the spectrum.  From those that blatantly tell us “never call me, just e-mail and don’t do that a lot” to customers who prefer a continuous level of communications using all of the communication methods available.
• Effectiveness - Are you performing the right tasks they need when they need them?  Just as you may find that you need to provide your customers with additional activities to support their needs, you may also be providing them with service activities that are not important to them.
• Quality of Support - Ask them to rate the quality of the services you are providing to them.
• Current Issues - Do they have any current issues that need to be addressed?
• Additional Information - What other questions should you be asking them?

As we learned in this blog, effective measurements are required to judge the success of any activity.  The quality of support you provide needs to be reviewed on a regular basis.  These questions allow your customers to provide you with important feedback on the quality of your support.  You can then “tune and tweak” your services accordingly.

Meetings should also be held with the customer groups that were asked to participate in the analysis.  It is important to make your customers aware of the external influences that may be affecting your support (inadequate staffing levels, time consuming projects, technical limitations of the environment).  These aren’t intended to be excuses but your customers may not know that some of their expectations will be hard to achieve until you provide them with the reasons.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Whether your customers are internal or external (like mine here at Remote DBA Experts), if you focus on just providing great technical support, you are only winning half the battle when it comes to keeping your customers happy.

One of the benefits that I am able to provide is that my organization supports dozens (and dozens……) of remote database administration services customers.   Our customer base spans the market spectrum from high technology to heavy industry.  In addition to having every market sector represented, the size of our customer base also varies widely.   We support organizations that have virtually no IT staff members to multi-national organizations consisting of huge IT staffs and hundreds of database servers.

This gives me the unique perspective of having a broad knowledge of what database support services customers want.   Name a market sector and I would be highly surprised if we didn’t have at least a handful of customers representing it.  Our customers’ technological strategies also vary widely.  Many of our customers don’t want to push the technological envelope while others want to stay at the forefront of every new database feature.

As I stated earlier in this blog, you need to be more than just a technician to be a success.   If you focus just on being a good technician, that is how you will be viewed and treated.   Being a mere technician or “table jockey”, as we call low strategic value DBAs at RDBAE, will not lead to a successful career path as a database administrator.   I’m responsible for service delivery at Remote DBA Experts.  When a customer views my organization as nothing but a pack of “table jockeys”, I have made a big mistake and have jeopardized that relationship.  If a customer sees us as a “utility provider” (no added value, just keeping the lights on), I’m in trouble.   In order to ensure our customer’s happiness, being a “table jockey” or a “utility provider” isn’t enough.  Providing excellent technical support is the foundation of a good relationship and building credibility, but it is only one facet.

Is this just a “remote DBA thing”?  Absolutely not!   You know that your customers expect you to be the technical expert.   If you aren’t at the top of your game technically, you will never obtain credibility with them.   But after you obtain that technical credibility – what next?   That will be the focus of my next set of blogs.  Will this require you to change your entire strategy? Once again, absolutely not.  Here’s a great quote that I often refer back to:

“Small changes can produce big results – but the areas of highest leverage are often the least obvious.”

- Peter Senge, The Fifth Discipline

But where do you start?   How do you determine how you are viewed?  Well, the first thing is you need to know where you are starting from!

It is obvious that effective measurements are required to judge the success of any activity.  In my next blog, we’ll discuss a DBA report card.  Questions that I have been using for years.  We’ll review some of the information you will need to gather in order to determine where you are when it comes to being a strategic-value DBA.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

This meeting is held as soon as the application developers are ready to begin the actual coding process. The ultimate goal of this meeting is for application developers to have a firm understanding of what is required to create and maintain a successful Oracle test environment. Discussions on Oracle utilities, SQL statements, procedural code, Oracle security, batch script testing, benchmarking, testing and monitoring are of prime importance at this time. The DBA must make it clear to all involved that the DBA unit is always available to answer questions and troubleshoot poorly performing SQL throughout the entire application development life cycle. It is imperative that the DBA make every effort to find and correct problems during this stage of the development process. Maintaining a proactive stance instead of a reactive one will always prove to be beneficial when it comes to correcting design problems. There are always enough potential problems lurking in the system for the DBA to solve without adding any additional ones through faulty application design. Topics to be discussed in this meeting include:

• Division of duties between DBA, systems, application development and business units
• Test hardware platform, O/S installation, configuration and administration
• Data feeds required from other architectures
• Data load and update processing and scheduling
• SQL Loader control cards
• Test system availability requirements
• Test system backup and recovery requirements
• Oracle security authorization
• Tools to aid in application development
• Oracle GRID toolset monitoring capabilities
• In-house monitoring tools
• Third-party vendor products
• Benchmarking and performance measurements

Monitoring Performance During Testing

Because this meeting and the previously discussed Setting Up a Successful Test System are closely related, they may be discussed together to save time and provide greater clarity. The DBA and developers need to have an understanding of what information is required to effectively monitor the performance of the Oracle application throughout the entire testing process. The DBA can offer to show developers how to use the various toolsets provided by Oracle (Oracle GRID AWR reports, SQL Trace, explain plan, autotrace, Oracle traces, V$ tables and my old favorite – Statspack) so that developers can play an active role in performance measurement. Suggested discussion topics are as follows:

• Names of all SQL Loader control cards and batch scripts
• Performance measurement goals and expectations
• Determine the test load volume (data volume, concurrent users) required to accurately predict production performance
• Comparison of test system load to estimated production load
• Oracle performance measurement tools. Suggested tools include:
  • Explain
  • SQL*Plus autotrace
  • Oracle SQL trace
  • Statspack
  • Oracle GRID monitoring components
  • V$ performance tables
  • Index utilization monitoring via ALTER INDEX MONITORING statement
  • Third party performance monitoring tools
  • LAN performance monitoring tools
  • Operating system performance monitoring tools

Performance Design Reviews

Information collected from the various performance monitoring tools is discussed at this time. One meeting may be sufficient, but large application development efforts usually require several discussions. If the DBA has maintained good communications throughout the initial stages of application design, there should be few surprises when these meetings are held. SQL tuning and tweaking recommendations are covered in this meeting. Depending on the length of the development process, follow-up meetings can be held to ensure that the application is performing as expected. Some suggested topics include:

• Load testing assessment: Is the load being placed on the test system large enough to accurately predict production performance?
• Review performance statistics collected during the testing process
• Assess SQL coding techniques by reviewing explain plan output for performance critical transactions
  • Index usage
  • Local/join predicates
  • Join methods
  • Subselects
  • View materialization
  • Determine if additional Oracle objects need to be created to enhance SQL performance
    • B-Tree Indexes
    • Bitmap Indexes
    • Function-Based indexes
    • Materialized views
    • Index organized tables
    • External tables

Preparation for Production Turnover

This meeting is held to determine if there are any last-minute questions and to make certain that everyone knows what is expected of them during the final turnover process. All units that have participated in the application design or application design review process are invited to attend. To ensure that all steps necessary for a successful migration to production are executed, the use of a standardized Oracle migration checklist is highly recommended. This document will allow the DBAs and developers to concern themselves with topics that are related to this specific turnover rather than spending time on the more mundane turnover tasks that are just as important, but easily forgotten. Having a complete, well thought-out migration checklist produces a less stressful and less error prone production migration process. Topics include:

• Division of duties between DBA, systems, application development and business units
• Production hardware platform, O/S installation, configuration and operation
• Data feeds required from other architectures
• Data load and update processing and scheduling
• SQL Loader control cards
• Backup and recovery
• Oracle security authorization
• DBA forms and turnover procedures
• Contact information and escalation procedures
• Post-production monitoring tools to be used

Post-production Turnover

This final set of meetings is held after the production turnover is complete. Application logic errors and performance problem resolution are the prime topics of discussion. A comparison of the actual performance to the anticipated performance of the application is also discussed. The review and correction process, by its nature, is iterative. The problems are identified, a plan of attack to solve them is agreed upon and additional meetings are scheduled to review the results. As additional problems are identified, they are added to the list of problems to be solved. This iterative process continues until all major performance issues and application logic errors are addressed. The post production turnover meetings should include discussions on:

• Review performance statistics collected (Oracle GRID, SQL Trace, explain plan, autotrace, Oracle traces, V$ tables and Statspack)
• Assess SQL coding techniques by reviewing explain plan output for transactions experiencing performance problems
• Index usage
• Local/join predicates
• Join methods
• Subselects
• View materialization
• Determine if additional Oracle objects need to be created to enhance SQL performance

Oracle Database Design Review Meetings – Conclusion

These recommendations are not intended to coerce readers into using the Oracle application design review meeting examples verbatim but to emphasize the importance of a structured approach to the design review process. The seemingly endless combinations of application architectures and software products used for application development may require the creation of a fully customized design review process for each application development project. The overall goal of design review meetings is to ensure the involvement of technical support units during the application’s design and implementation. When Oracle design issues are addressed early in the development lifecycle, problems are minimized and the migration from test to production is more easily accomplished.

Check out The Art of Being a Successful DBA – Application Design Review Meetings, Part 1.

Thanks for reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

One of the overlooked duties of an Oracle DBA is to inform, educate and assist the application development staff during the application development process. Although these responsibilities may not be formally assigned to the DBA, the DBA unit often finds that they are providing these services by default. The DBA is often considered to be a seasoned veteran who spends most of their time learning the subtle eccentricities of the database management system.

It is the DBA’s responsibility to ensure that the overall design and implementation of the new application is proceeding according to expectations. Although application developers may be experts in SQL, procedural languages (Java, PL/SQL, C variations, TSQL, etc.), they still turn to the DBA for help with standards, procedures, performance design, error recovery and complex SQL.

A continuous and effective dialogue between the DBA unit, system support personnel and application developers is of utmost importance throughout the entire application design and implementation process. One method to foster effective communications is for the DBA unit to create a series of new application design review meetings. These meetings can be organized in a manner that promotes efficient application usage of the Oracle database environment. The design review meetings can be scheduled during logical break points between the different stages of the application development process.

The database administration team should invite representatives from the development and business units to assist in the creation (and enforcement) of the standardized review process. Application development teams will be able to help tailor the design review process to satisfy their specific design and implementation needs. Business users who will be impacted by the production turnover will provide valuable input on implementing new applications for their specific business areas. Customized checklists are created to ensure that all application and business specific issues are addressed during each meeting of the design review process.

It is recommended that the database administration team then communicate to all application development and business areas that any internal applications created without following the standardized review process will not be migrated to production. Having the application teams and business units participate during the creation of the standardized review process allows the DBA team to enforce the policy without being viewed as dictatorial or making rules in a vacuum. In addition, every effort should be made to “sell” the standardized design review process in all communications to application development teams and business units.

The following list of meetings can be used as a starting point in the creation of a structured application design review process:

Initial Overview of Proposed System

The kickoff meeting is held with datacenter operations support managers, operating system administrators, application team leaders, system analysts, end-user management and data administrators to discuss the application’s general purpose and function. This information will allow support technicians to determine the impact the new application will have on existing systems and allow them to begin planning the application design implementation process. The following information should be covered in this meeting:

• General description of purpose and function
• Application scope (enterprise wide application that affects all business units, intra-departmental that affects several business units or departmental)
• General application size (estimated number of programs, data objects)
• Criticalness of the application (mission critical, application critical, non-critical)
• Application availability requirements and downtime windows
• Application type (decision support, business intelligence, data warehouse, data mart, online transaction processing)
• Architecture design (web server, application server, N-tier, distributed database environment)
• Advanced features required (replication, advanced queuing)
• Data feeds required from other architectures
• Load processing
• Batch processing
• Online transaction processing
• Development tools used to build front-end application screens
• Third-party tools used to provide Ad-Hoc query access
• Procedural language used to build business logic (Java, PL/SQL)
• Application development methodology to be used (Agile, Waterfall, RAD)
• Number of concurrent users
• Disk storage and data growth estimates
• Highly available architecture discussion (RAC, Oracle Fail Safe, Data Guard, hardware vendor clustering and failover)
• Performance expectations
• Criteria used to judge performance
• Security and auditing requirements
• Hardware platform, O/S preferences/selection and sizing discussion
• Hardware platform, O/S installation, operation and administration
• Division of duties between the DBA, application development and business units

Logical Data Model Review

This meeting is convened as soon as the logical data modeling effort is finished. The major emphasis of this meeting is to determine if the logical data model is complete and correct. The application’s process model (if one is available) can also be verified at this time. Volume statistics, object growth rates, purge criteria, referential integrity needs and application-naming conventions are also discussed. Knowing your data before hand is essential to designing processes to manipulate that data. The following topics are covered in this meeting:

• Determine if the data model is fully documented (entities, attributes, relationships)
• Attributes have correct datatype, length, NULL status, default values
• General discussion of business rules that are to be enforced by database level constraints
• Not null constraints
• Check constraints
• Unique constraints
• Primary key constraints
• Foreign key constraints
• Business rules to be enforced by triggers and procedures
• Business rules to be enforced by application code
• Logical/process model comparison
• Volume statistics
• Growth rates and purge criteria

Designing for Performance

This meeting is held with the application development units before any physical DDL is generated by the DBA. Proper transaction design and efficient SQL coding results in less performance-oriented database alterations made during the latter stages of the design and implementation process. Determining predicate usage will allow the DBA to create additional database objects (indexes, materialized views, index organized tables) to increase SQL, and subsequently, application performance. The following information is discussed:

• Normalization vs denormalization or “lets collapse those 49 tables into one big one for fast read access”
• Table access requirements and predicate usage
• Database objects used to increase SQL performance including:
  • B-Tree Indexes
  • Bitmap Indexes
  • Function-Based indexes
  • Materialized views
  • Index organized tables
  • External tables
  • Data partitioning algorithms (range, hash, list, etc.)
• Process parallelism (parallel query, parallel DML, parallel load)
• Transaction modeling
• Oracle SQL performance features
• Full table scan vs index access
• Hash joins
• Star joins
• Index skip scans
• Index fast full scans
• Cursor sharing (SQL statement reuse and soft parse vs hard parse)
• Bind variables
• Reinforcement of effective SQL coding techniques

Read The Art of Being a Successful DBA – Application Design Review Meetings, Part 2.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery