In last week’s post, I started providing a general set of recommendations on keeping your customers happy. Today, I’ll continue our discussion on “The Art of Being a Successful DBA” with my remaining recommendations for customer satisfaction.

Application Design Reviews
Yet another activity that I firmly believe in.  Just call me “Mr. Procedure”…  I’ve just seen way too many things go wrong in 20 years in this profession to not take a proactive problem prevention approach to the development process.   The bigger the application being developed, the more proactive you need to be.

I used to be a “high priced industry consultant” that customers would pay a fairly substantial rate for.  As a result, I would be brought in to solve the “tough problems”.   What I found was that if these customers WOULD HAVE FOLLOWED ANY TYPE OF DESIGN REVIEW PROCESS THE MAJORITY OF THESE PROBLEMS WOULD *NOT* HAVE OCCURRED.  Ok, now I’m yelling.

I was once asked to be onsite for a huge application turnover for a local business.  I had about 4 hours notice to drive out to a firm and help the other consulting company when they “flipped the switch” on the new application.   Although this consultant group was from the largest technical firm in our industry, documentation (and application design best practices) was dismal at best.  I just had a very uneasy feeling.  This application was BIG.  I could also tell my fellow onsite DBA was not very experienced and very overwhelmed.  He stated that most of the DBA work was done offsite.

Well, when “they flipped the switch” – “the lights went out”.    Nothing, and I mean nothing, wanted to run.   I looked at the other consulting DBA, his eyes were as big as pie plates.  He looked at me, said “I quit” and began to pack his equipment up.   I whispered to him “don’t ruin your career over this, let’s take a look first”.

Sky-high CPU was the culprit.  The next step was to look at some diagnostic information (we’ll be covering this in-depth in upcoming blogs).  Reviewing the SQL provided in a SQL Trace dump, I noticed nothing, and I mean nothing, was using bind variables.  This application also had dozens of batch jobs that processed around the clock.  The jobs were looping and hard parsing, looping and hard parsing – statement after statement after statement.

I thought to myself “this may not be another 36 hour, around-the-clock gig”.  The database went down, the cursor sharing parameter was changed, and the database brought up.  It was like we switched to the Energizer Bunny.  The application just ran.  I thought “wow, I never knew hard vs soft parsing made that much of a difference”.    The culprit was that this very blue consulting organization used a new fangled code generator that didn’t generate bind variables.

I have been involved in dozens of cases like these.  When I asked if anyone went through a performance review during the design process, I received blank stares.  The moral of this, and many other horror stories, is that they *ALL* could have been prevented if any type of design review process was followed.

The goal of the design review process is to identify and address application design, process flow, program logic and SQL statement problems early in the development life cycle. Identifying these issues early in the development life cycle allows them to be more easily addressed than if they were to be identified during later stages.  That last statement bears repeating.   Like any other issue you face, no matter what it is (personal, professional, whatever), the sooner you identify it, the easier it is to correct.

Over the years, I have come up with a set of design review meetings that any DBA unit can follow.  These recommendations are not intended to coerce readers into using the application design review meeting examples verbatim but to emphasize the importance of a structured approach to the design review process. The seemingly endless combinations of application architectures and software products used for application development may require the creation of a fully customized design review process for each application development project. The overall goal of design review meetings is to ensure the involvement of technical support units during the application’s design and implementation. When design issues are addressed early in the development life cycle, problems are minimized and the migration from test to production is more easily accomplished.

DBA Report Cards and the 360-Degree Review Process
Effective measurements are required to judge the success of any activity. The quality of support the DBA unit provides should be reviewed on a regular basis. A DBA report card allows business and application development units to provide feedback on DBA support activities.

The report card will allow them to measure how well they feel you are meeting your Service Level Agreements.  Here are a few sample questions to start you on your way:

• How would you rate the turnaround times for DBA unit work requests?
• How would you evaluate the DBA unit’s responsiveness to questions?
• How would you evaluate the DBA unit’s responsiveness to requests for assistance?
• Please rank the quality of communications the DBA unit provides.
• Please rank the overall quality of work the DBA unit provides.

All questions are ranked from 1 to 5 with 5 being the highest. You can also allow the respondents to rank the importance of each question from 1 to 5 with 5 being the highest.

A short survey should be attached to the DBA report card to gather additional information that can be used to improve the quality of support the DBA unit provides to their customers:

• What are your top three technical challenges that you face?
• What are the top three non-technical challenges?
• Please list your current priorities. Rank them in order of importance.
• List the most important services the DBA unit provides. Rank them in order of importance.
• What support services does the DBA unit do a good job of providing?
• What support services should the DBA unit improve upon?
• What additional services would you like the DBA unit to provide?

Meetings can be held with the respondents to discuss their reviews. DBA team members participating in the reviews must be prepared to respond to criticism in a professional manner. But just as its title describes, the 360-degree review process also allows support units to provide feedback on their customer’s support requests and work activities. The 360-degree review process provides important feedback to both support units and their customers. Once again, your customers may not know that some of their expectations are unachievable until you tell them the reasons why.

Corrective Action Reports
Databases are challenging to administer. As much as we would like to prevent mistakes, we do make them. That’s one of the benefits of being human; we learn from our mistakes. During my 20 years in this field, I’ve seen all kinds of mistakes made – from little ones to catastrophic. I’ve also made my fair share of them.

The worst one I ever experienced is when we brought a technician in from a third-party disk vendor to format what we thought was to be a single disk in a huge disk array. We just purchased the arrays and were in the process of assuming support responsibilities from the vendor. The vendor assisted us in the initial setup and data migration from our old storage devices to their new whiz-bang storage system.

The array stored data from dozens of Oracle databases. After 20 minutes of showing our folks how to set the necessary switches and enter the appropriate values at the prompts, the technician hit the enter key with much flair and bravado.

I assumed we would see the one activity light activate for the disk being formatted. Instead, I saw a whole wall of lights begin flashing and twinkling. I thought to myself “Hmmm, this can’t be good…”. I glanced at the technician and the look of horror on his face confirmed my suspicions. He then reinforced my conclusions when he stammered, “I think I just formatted the entire array.” I looked at the DBA next to me and said “Looks like we are in for a looong night.”

I continue to be amazed at the rapid improvements in hardware redundancy and reliability. Hardware platforms now have the ability to recognize individual component failures, bypass them, produce diagnostic information and “phone home” to report the problem to the manufacturer. But hardware components will fail. Just as the sun comes up every morning, hardware components will fail, operating systems will get tied up in knots and database bugs will manifest themselves. Such is life for a computer technician.

The key is to document everything that happened during the time period when the problem occurred. Documenting provides you with the information you need to prevent the problem from occurring again and your customers with information on exactly what happened. Its my experience that the more informed a customer is about the problem and the steps your unit is taking to prevent its reoccurrence, the better they feel.

The corrective action report should contain:

• A detailed description of the error that caused the problem.
• The customer impact the problem caused.
• A timeline of the activities that were performed.
• The steps that were performed to correct the problem.
• Mitigating factors that contributed to or exacerbated the problem’s impact.
• The steps that will be taken to prevent the problem from occurring again. Include who is responsible for the activity and a date the activity will be completed.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Being a DBA is more than just being technical.  Over the years, I’ve found that becoming lax in non-technical areas of database administration and not following my own best practices could create just as much havoc as a technical problem.

I don’t care what database you support.  If you want to excel in this profession, you need to be viewed as more than just a DBA “table jockey”.   You must be seen as someone who understands both the business and technical aspects of the applications they support.  Gone are the days when being technically proficient was enough to get by.   That is definitely not the case in today’s business environment.

Let’s start finishing up this series on “The Art of Being a Successful DBA” with a general set of recommendations on keeping your customers happy.

Good Communication is the Key to Successful Customer Relationships
As most readers know, I run a remote database administration services shop in Pittsburgh called Remote DBA Experts.   It is a tough business.   Customers have turned over their most valued data assets to us.   It is a responsibility that we don’t take lightly.  We are held to a very high standard by our customers and that’s OK with us.  We should be.

Achieving “0 technical defects” goes without saying.   We have virtually dozens of in-house procedures, checklists, etc. to ensure that our work is of the highest quality.  But we must also make every effort to ensure that effective communications are occurring between our organization and our customers.  Not having the luxury of “face time” with our customers on a daily basis drives us to use all communication mechanisms possible to ensure our customers are comfortable with the services we are providing to them.

But the same high-quality communications requirement holds true with your internal customers. You need to create and foster good communications from the very beginning of your relationships with end-users and developers.  From internal Service Level Agreements to project statuses and problem notifications, DBA units need to maintain constant, high-quality communications with their customer base.

Here’s a quick laundry list of hints and tips that can jumpstart your strategy to improve communications with your customers:

Document and Distribute Service Level Agreements (SLAs)

Service Level Agreements (SLAs) help to solidify support requirements and dispel any inflated expectations a business or application development unit may have.  They probably won’t be aware of your current workload and resulting work request lead times until you tell them.   A standard set of work request completion timelines should be established for the organization.  The work requests can be categorized any number of ways.   Here at Remote DBA Experts, we have standard documentation that breaks the work requests into categories based on complexity and the estimated time to complete them.   We then provide numerous examples of activities that fall into each of the categories.

The DBA team lead should also meet with each individual unit supported to establish a set of measurable Service Level Agreements that include support activities required, maintenance windows, and application performance and availability objectives.  A unique customer profile for each unit supported can be created that describes everything the DBA unit needs to know to provide high-quality support.

The notification lead-time required for the database administration team to perform a requested change can then be formalized, documented and distributed to business and application development units.

DBA unit customers need to understand the amount of time the DBA unit will take to process a given work request.  This will prevent your team from getting a request to migrate a database from test to production in the morning with a required date for that afternoon.  Of course we all know that never happens.

But we also understand that the world does not revolve around the DBA Unit.  I don’t care which business any of our customers are in.  Each market sector is a competitive one (except maybe if you’re in government).   There are times when business requirements require that we “burn the midnight oil” to get our administrative work complete.  That is part and parcel of what DBAs do.  I was in a conversation with a customer recently.  Although I kept telling him he really had no need to apologize for the rash of short-term requests he was asking for, he stated, “Chris, it’s not like I’m sitting on these until 5 minutes before they are needed.  We are trying to remain competitive and the business users think this application will help us survive.”

Document and Distribute DBA Unit Notification Methods

How do you want your customers to contact you for work requests?  Is it a traditional ticketing system, e-mail or phone call?  Your unit’s ability to react to a work request, problem or outage notification depends upon the mechanism used to notify you.

• Support Line – Support lines don’t have to be for external customers or off-hours calls only.  An internal support “hot line” can be used to notify the DBA unit of problem during the day as well as after hours.   The hot line can be used for any issue that affects the availability of the database.   This includes outages, global performance issues, data quality issues (data changed/deleted by mistake) and connectivity problems.  The support line should also be used for emergency change requests that must be performed immediately.
• Support Tickets– If your unit uses a ticketing system, you need to provide hints and tips for your customers to ensure miscommunications do not occur.   What are the procedures if one of your customers enters a ticket after your unit’s workday is complete?  How about weekend tickets?  If you assign tickets entered after 6PM on the morning of the next work day, you need to let your customers know that.  The same is true for weekends.  How about work request priorities?   If you don’t tell your customers what a critical priority means in terms of turnaround time, you have a very good chance of miscommunications occurring.   You need to be crystal clear on virtually every piece of information that is associated with that ticket.    In this case, too much is probably still not enough.                                                                                                                     
• E-mail Groups and Individual E-mails – Probably the most common and least preferred method of communicating with any support unit.  It is important to tell your customers that an individual technician may be out of the office for any number of reasons (vacation, illness, business travel, etc.).   All support technicians should be required to configure “out-of-office” greetings for their desk phones and e-mail accounts when they are out of the office for any reason.

Internal DBA Unit website
Provides your unit name, mission statement, SLAs, databases supported, personnel contact information, calendar listing DBA vacations, on-call, etc.  The more creative you are, the more popular your internal website will become.

Application Change Management
Database administrators usually support different business units with each unit having their own set of unique procedural requirements. Formalizing and documenting the change request process minimizes the potential for miscommunication between the business units, application development areas and the database administration unit. This is even more important to remote DBA services providers.  Standardized communication practices are a key ingredient of a trouble-free application environment.   This ensures that there are no “surprises” when a change is implemented in production.

If your organization doesn’t have a formal change request process in place (and many shops don’t), create your own! There are dozens of change management and source code versioning tools available on the market today. The prices can range from thousands to tens of thousands of dollars.

Although I highly recommend these types of products, I wouldn’t let the lack of having one prevent me from formalizing the change management process. You do the best with what you have.  I have an entire blog devoted to application change management best practices.

From creating a single index to the implementation of a complex database-drive application, good communication is absolutely critical during the change management process.

Wrapup
I hope you enjoyed the first part of my recommendations on keeping your customers happy. Look for the conclusion in next week’s post.

Thanks for reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

In an interview with Bill Gates and Warren Buffet at the University of Nebraska, Buffet was asked what one piece of advice he would give to others wanting to be successful. He stated that students “should polish their public speaking skills”.

As I wrote in a previous blog, take a look at your last performance appraisal forms, I’m betting that a lot of the criteria you are being judged upon depends upon communications. The key words and phrases to look for are “ability to work in a team environment”, “keeps supervisors informed”, “maintains good communication with the user community”, “ensures the content of the communication is at the appropriate level for the intended audience”, “provides system and user documentation for projects and system enhancements”. I pulled all of the aforementioned phrases verbatim from one of my own past performance appraisals. I reviewed all of the criteria that I was being evaluated upon and found that almost seventy percent of the items depended upon verbal or written communications.

So, let me get off my soapbox and get to the topic at hand. How exactly do you improve upon your verbal and written communications? If you don’t have good communication skills, all is not lost. Like anything else, these skills can be learned.

Improving Verbal Communications
We have all heard that speaking in front of an audience is the number one fear for most folks, surpassing both financial and health problems. Let me give you a couple of hints and tips that have helped me. Although, I still don’t view myself as an accomplished speaker, I’m a LOT better than I used to be.

Toastmasters International – I can not recommend this organization highly enough. Toastmasters International is a non-profit organization that focuses on improving communication and leadership skills. Toastmasters is also a great place to network and advance your career (it makes for an impressive bullet on your resume).

Toastmasters International consists of hundreds of Toastmasters Clubs that are governed by a Board of Directors that is elected by Toastmaster members. The Toastmaster website provides a search function that allows visitors to find nearby clubs. If you want to learn how to speak effectively in public, do yourself a favor and search for the clubs in your area. I think that you will be surprised at how many clubs there are.

Everyone that attends a Toastmasters event is there for one reason – to improve their public speaking and leadership skills. You are not only taught the skills you need but you are also provided with the opportunity to showcase your new found talents by giving various speeches to fellow members. The key benefit of speaking at Toastmasters is that you are giving speeches to others in a supportive environment. Everyone there wants to improve their communication and leadership skills. A great place to become an accomplished speaker.

Formal Classroom Education
It would be a rare event indeed if you found an institution of higher learning that didn’t provide classes on public speaking. If you are attending school to obtain that next degree, you need to make sure one, or more, of those classes pertains to verbal communications. The more the merrier. Many colleges offer adult education curriculum at night. Review the night class curriculum for local colleges. I would be surprised if you didn’t find one or two classes on public speaking.

Books
This list will be short. I am an avid fan of anything written by Dale Carnegie. My all time favorite book from this author is titled The Quick and Easy Way to Effective Speaking. Amazon carries this book in stock. The benefit it provides far outweighs its current price tag of $7.99. If I could only have one book on public speaking, it would be this one.

Personal Recommendations
Here’s a quick list of recommendations that have helped me improve my own public speaking skills:

• When you attend presentations from others, pay attention to both the material they are presenting and how they present it. Two of the most engaging speakers I have ever had the pleasure of meeting are Craig Mullins and Richard Niemiec. I actually attended one of Niemiec’s presentations a second time. The first time I attended was to learn the material he was presenting and the second time I focused on the techniques he used to engage the audience. I once saw a representative from Microsoft give a presentation on their new operating system to a group of 700 people. By the end of his speech, he had the entire audience “high-fiving” each other on a regular basis. I thought to myself, “he generated this much enthusiasm over an operating system?”. I stayed right where I was and watched him give the presentation to a second group of 600 with the same result. The second time I paid close attention to his presentation style and stage mannerisms. I’m not saying that you should attempt to clone your presentation style from a particular speaker, but you can improve your own communication skills by learning from accomplished speakers.

• The more you speak in front of an audience, the better you will become. Absolutely, positively no doubt about it that “practice makes perfect” in this case. Start your public speaking career by starting with a small audience. Maybe you know a particular database feature or a tip or trick that you think your fellow technicians may benefit from. Invite them to a meeting, create a presentation and present it! Work your way up to larger audiences. Join local users groups and ask if they would allow you to give a presentation. Every time there is a call for presentations from IOUG or Open World, I submit several presentations. I may not get picked all of the time but I still try.

• I have found that the more prepared I am, the less I am affected by anxiety before and during a speech. I will present a speech to myself, my wife, my dogs numerous times before I give the speech in public. The more times I do it, the more confident I am.   I always visit the room I am going to speak in. It makes me more comfortable when the time comes to stand up in front of the audience. I also stand right in the middle of the aisle and introduce myself to as many participants as I can. It helps me to feel more comfortable when I step up on stage.

• It pays to learn from your mistakes (and have a sense of humor about them as well).  I had a previous speaker at Open World walk off the stage with a tie mike I was supposed to use.  The stage hands couldn’t find the previous speaker to retrieve the tie mike or find a replacement in time.  What they did find was a corded 3 foot tie mike.  I saw the length of the cord and though I would be OK if we could plug it in somewhere near the podium.  Problem was the only plug available was right in the middle of the floor.  I gave a 1 hour presentation bent over at the waist.  I also learned that you should never drink a carbonated beverage wearing the newer, more sensitive tie mikes.   They’ll pick up every little sound that a carbonated beverage makes.   Lastly, always check the podium to make sure it is stable before you lean on it.  I leaned on one and quickly found out it was on wheels. Of course those wheels were unlocked.   I leaned on the podium and quickly went careening towards the end of the stage.   People in the first row had the same expressions as the folks that were just about to get stepped on in old Godzilla movies.

Improving Written Communications
Have you ever read a document or e-mail that was so poorly written that you had a hard time understanding what the writer was trying to convey? What was your opinion of the author? Not good, I’ll bet. Whether we like it or not, we will be continuously judged upon our written communications throughout our careers. From entry-level technician to CEO, you will be judged by the words you write.

You’ll find that my recommendations to improve your written communication skill sets closely parallel my public speaking recommendations. They focus on formal education, gaining experience and best practices.

Formal Classroom Education
Universities, colleges and high schools all offer adult night classes. Like public speaking courses, you will be able to find a local educational institution that offers classes on effective writing skills. I still have a couple of my college textbooks on my desk that I use as references.

Books and Websites
You need the tools of the trade to write effectively. As I stated previously, I still use a few of my aging college textbooks as references. I also have Anne Stillman’s book appropriately titled Grammatically Correct: The Writer’s Essential Guide to Punctuation, Spelling, Style, Usage and Grammar. This is the book I refer to about 90% of the time when I am attempting to improve my grammar, spelling and punctuation.

I also visit one website regularly when I write. If you hop on to your browser of choice and navigate to http://www.thesaurus.com/ or http://www.dictionary.com/, you will go to the same website. This very helpful website allows me to quickly find definitions, synonyms and antonyms. If I want to obtain information specifically on grammatical rules and concepts, I’ll visit the Guide to Grammar and Writing website. The numerous drop-down lists allow me to quickly navigate to the topic I am looking for.

You can also take advantage of Microsoft’s Spelling and Grammar Checker tools in Word. You will need to be careful with some of the recommendations. Use Anne Stillman’s book or the Guide to Grammar and Writing website as a reference if you suspect that the Microsoft tool’s recommendation is incorrect.

Obtaining Assistance from Accomplished Writers
When I first started working in a corporate environment, my writing skills were terrible. But I had the good fortune of having a manager that understood the importance of both verbal and written communications. I would write a memo, she would correct it with her red pen and send it back to me for a rewrite. Many of them had a “Nice Try!” and a smiley face on top. Even though all of the rewrites (and smiley faces) were somewhat exasperating, her persistence forced me to become a better writer.

I also had the good fortune of being able to use a very accomplished writer, Craig Mullins, as an editor during the initial stages of my writing career. His first piece of advice stays with me to this day. I told him that I often get writer’s block when I first start. He stated “Hey, who says you have to start at the beginning of the article? After you do the initial article layout, start anywhere you want to. If your initial design and layout are good, it will be easy to tie it together.”

It’s relatively easy to find fellow technicians that excel at written communications. Think about all of the e-mails and documents that come across your desk on a daily basis. You can read the content and also review the text for ease-of-reading and grammatical style. Find the folks that are good at written communications and ask for their input. I think you’ll find that most of your requests will be warmly received.

Practice Makes Perfect
Like public speaking, the more you write, the better you’ll get. That’s one of the benefits of writing this blog. It keeps my skills sharp. When I first started this blog, I would often agonize over every written word. Although I still get stuck from time to time, the words now flow a little more easily for me.

You also need to become involved with as many company newsletters and related communications as you can. When I asked to join a newsletter, I always started with “I’m not the greatest writer, but I’m trying to learn.” I also asked my peers that worked on the newsletter to critique my work. The more I was critiqued and the more I plugged away at writing, the better I became.

Wrapup
IT shops are no longer evaluating technicians purely on their technical skills. It is the total package of skills that you bring to the table that you are being evaluated upon. We all know the importance that our technical skill sets have upon our success in this field. But you also need to be well rounded in all of the skill sets your managers are looking for.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

“The hallmark of a good DBA is not knowing everything; it’s knowing where to look when you don’t.”

I thought it might be beneficial for us to review some of the various technical resources currently available to us. You’ll find a listing of my favorite authors, bloggers and websites in this blog. The list will also include my favorite educational websites of all time, Oracle Technet, the Oracle Knowledge Center and Oracle Metalink.

I have a LOT of books on Oracle. I am an avid Oracle book collector. I buy them, publishers send them to me to review, and I receive them as presents from co-workers and family members. I have been working with Oracle for over 22 years now. I have been using Oracle since Version 5 and I have kept every Oracle book, manual and instructor’s guide I crossed paths with during that time. Remember the old paperback manuals that Oracle sent to you with the software? I’ve got LOTS of them! I was certified by Oracle to teach virtually the entire DBA admin track and I have kept every version of every instructor’s guide I used (Version 7 to Version 10). As a result, my library consists of dozens and dozens (and dozens) of books on Oracle.

There is so much information available on Oracle that it tends to become overwhelming. How do you find that one facet of information, that one explanation you are looking for when you are confronted with seemingly endless sources of information? Here’s a hint, GO TO THE MANUALS FIRST.

Call me old school, but I still read the entire set of documentation for every new Oracle release. The order of the first few books is always the same: “New Features”, “Concepts”, “Administrators Guide”, “Reference”, “SQL Reference”, “Performance Tuning Guide” and “Data Warehousing Guide”. The remaining books are read in no particular order. It’s pretty much what I feel like reading at the time. When I was teaching, I always reinforced to my students that they needed to reference the Oracle manuals before moving on to other forms of documentation. I’ll read the “New Features” from cover to cover. For some of the others, it isn’t so much reading than scanning for changes.

A very experienced co-worker of mine was at a customer site installing an Oracle database on LINUX. He was reading the installation manual when the customer demanded to know why he was reading the manual when he was supposed to be “the high-priced expert.” He quickly replied “I’m reading the manual because I am an expert.” As your experience grows, you’ll find that you’ll become just like my co-worker, an avid user of the reference guides and not afraid to admit it.

Reference Manuals
I have seen many mistakes by DBAs picking the first release of the reference manuals they find on Technet. It is a much better practice to use the set of reference manuals for the release of the database you are administering. Oracle does have a tendency to change default values for object specifications. In addition, each new release contains new parameters that affect the database’s configuration. When you receive the latest and greatest version of Oracle’s database (one of the benefits of purchasing support), turn straight to the “OracleX New Features” section to find out what impact the new release will have on your daily administrative activities. You’ll also find many new features that haven’t been covered by Oracle’s new release whitepapers and marketing propaganda. I always find a half dozen to a dozen little-known features that aren’t widely publicized but very important to my organization.

Oracle Internal Resources
The Oracle websites contain a wealth of information on the Oracle product sets. The following Oracle websites are favorites of mine:

metalink.oracle.com - Oracle’s premier web support service is available to all customers who have current support service contracts. Oracle MetaLink allows customers to log and track service requests. Metalink also allows users to search Oracle’s support and bug databases. When you experience an Oracle problem, look up the return code (if one is provided) in the Oracle reference manuals. If you are unable to solve the problem, search Metalink using the return code or error message as the search criteria. The website also contains a patch and patchset download area, product availability and life-cycle information and technical libraries containing whitepapers and informational documents.

I’ve been able to solve 80% of my problems using the search feature. Here’s a couple of tips: make sure you specify “ALL SOURCES” when searching for problems, use the Advanced feature to narrow your search to specific products and remember that Oracle now adds a listing of products and asks you if you want to limit your search to them at the top of each search result set.

While you are on Metalink, click on the “Knowledge” tab. It will display the Oracle Metalink Knowledge Page. You will find that this page is a hidden treasure trove of information. You will need to drill down into the different sections under the “Knowledge” tab to retrieve all of the gems of information available. My personal favorite is the Tools and Training page. You need to spend some dedicated time on this section. The information here is priceless to those seeking to improve their Oracle knowledge. There are also 300 scripts available to choose from!

technet.oracle.com - Technet’s software download area allows visitors to download virtually any product Oracle markets. This website stores technical reference manuals for Oracle database releases 7 through 11, Oracle RDB, Oracle Gateways, Oracle Applications, Oracle Application server – the works.

The site contains links to many Oracle blogs  and has forums, sample code and tutorials. A quick and easy way to get access to the information you need. Visitors are also able to view Oracle discussions, download product whitepapers and obtain information on Oracle education. Can you tell that I love this site? YOU BET!

partner.oracle.com - If you are an Oracle partner, (and there are a lot of them), then this is the website for you. Oracle’s partner website contains information on partner initiatives and provides customized portlets categorized into partner activity and job role.

education.oracle.com - Oracle University’s web site contains information on Oracle education including course descriptions, class schedules, self-study courses and certification requirements.

When you go to education.oracle.com, look on the left side of the screen. You will see a link to the Knowledge Center. The Knowledge Center is Oracle’s online education portal. You can take dozens of online courses from Oracle’s experts. Want to learn from the people that wrote the code? This is the place that will allow that to happen.  The knowledge center states that it now offers over 3,000 online courses.   Well worth the fee Oracle charges.

There is a link on the right side of the Knowledge Center that says “Free Trial”.  Do yourself a favor – TRY THE FREE TRIAL . You will have to pay for a full-blown subscription, but the benefits the Knowledge Center provides are well worth the cost. Take the free trial and then weasel the money out of your company for a full subscription. Tell them you’ll work late, wash the boss’s car. As I said, it’s well worth it.

www.oracle.com - Oracle’s home page on the web. Contains white papers on Oracle features, product descriptions, etc.. Good place to get a high-level overview of the Oracle database’s bells and whistles.

External Resources
Non-Oracle websites are also excellent sources of information. The Internet has an abundance of web sites containing hundreds of scripts, tips, tricks and techniques. I often find myself heading straight to Google when I can’t quickly find the information I need on Metalink. Some of my favorites are:

asktom.oracle.com - Kyte maintains an Oracle Q&A website aptly titled “Ask Tom”. If you scan the Ask Tom website, you’ll quickly find that his primary method of simplifying complex technical concepts is to use a snippet of code as an example. Based on the positive responses he receives, the code examples work. They also work for me too. A great website.

jonathanlewis.wordpress.com - The Oracle Scratchpad by Jonathan Lewis contains his blogs, related musings, etc.. You can’t go wrong with the information that Jonathan provides. Two of my favorite websites are the Scratchpad and Ask Tom.

www.orafaq.com - Orafaq discussion forums are excellent sources of information. Post a question to hundreds of experienced Oracle DBAs and you’ll find out just how helpful Orafaq can be. Orafaq provides an intelligent search engine that visitors can use to search the discussion forums for topics of interest. The website also provides hints, tips, scripts and whitepapers.

www.oraclecommunity.net - I like this site because it provides you with more than just a dump of technical information.  It allows visitors and members to communicate with each other and discuss topics in an informal manner.    People introduce themselves to each other, exchange ideas, and inform others of upcoming events.   Nice site.

www.oracle.com/oramag – Oracle Corporation’s own technical magazine. Oracle Magazine provides readers with product announcements, customer testimonials, technical information and upcoming events. Oracle magazine is available in hardcopy and on the web.
Blogs and Wikis
What I like about blogs is that their free-form style makes them interesting to visit. Some blogs contain small snippets of information, while others are more like articles. I’ve always been grammatically long-winded by nature, so my blog is often longer than most. Wikis are collections of information that can be added and edited by the internet community. They are similar to a blog in structure but can contain the works from many different contributors.

Blogs that I like are:
wiki.oracle.com/page/List+of+Oracle-related+blogs – Not actually a blog, but a Wiki that contains dozens (and dozens) of different blogs. If you can’t remember that blogger’s name, you should be go down through the list and hopefully it will jog your memory

www.petefinnigan.com - Pete Finningan’s Oracle Security blog and general Oracle information. More than just a blog, the website has whitepapers, discussions, scripts, Oracle news….

richardfoote.wordpress.com - Not just because his last name is like mine. The guy is an index and performance GURU. Lots of mini presentations to choose from.

feuerthoughts.blogspot.com - Oracle PL/SQL language. I’m absolutely terrible at coding anything (all that IF-THEN-ELSE-OTHERWISE-GOTO-EXIT stuff drains me). I go to his site to attempt to improve my PL/SQL skill sets. If his teachings can’t help, no one’s will. Excellent writer and creator of some pretty neat tools. Make sure you traverse the links to his various PL/SQL pages. Tons of great information.

Book Recommendations
Third-party books are another excellent source of information. The big advantage third-party books have over the technical reference manuals is that technical reference manuals must provide all of the information on the entire Oracle environment while third-party books are able to focus on just what the author felt was important. If you choose a good author, you’ll learn quickly.

Instead of listing books, I’ll provide you with my favorite authors. You’ll have the best chance of buying a great book if you select one from the following authors: Thomas Kyte, Jonathan Lewis (his book titled “Cost Based Oracle Fundamentals” is one of my favorites), Arup Nanda, Craig Mullins, Michael Abbey, Michael Corey, George Koch, and Kevin Loney.

Oracle Press Books provides a lot of great Oracle titles and Appress Publishing also comes to mind as having a great set of Oracle literary offerings.

Wrapup
I hope you enjoyed this brief discussion on where to find Oracle information. Don’t attempt to remember everything about Oracle just try to remember the best place to look when you need information. It’s out there!

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

I started my career working in a mainframe environment. Fellow blogger Craig Mullins and I administered DB2 databases on huge, big iron platforms. Platforms that supported thousands (upon thousands) of concurrent users. One of the benefits of having a background in mainframe support is that it taught us the importance of change management best practices.

Craig and I learned that a key ingredient of a trouble-free mainframe environment is ensuring that there are no “surprises” when a change is implemented in production. Changes that can affect the day-to-day business operations of an entire organization. Throughout my career, I have applied these “mainframe style” best practices to all other database ecosystems that I was responsible for supporting.

It works. The first company I applied these best practices to was selected by Oracle as a “Showcase Environment”. This was back in the old days, when Scott’s tiger was just a cub. Oracle identified shops that it thought had rock-solid support environments and asked them to host visitors from other organizations that wanted to create their own high-quality support infrastructures.

We apply these same principles to our remote service offerings here at Remote DBA Experts. We support hundreds of developers around-the-clock. If a customer does not have a strong set of application development change management procedures in place, we will help them create one during our transition activities.

I thought I would provide you with a few helpful hints and tips on change management best practices.

Database Design Reviews
One of my earlier blogs provides information on database design reviews. Database design review meetings foster effective communications between the DBA unit, system support personnel and application developers throughout the entire application design and implementation process. When Oracle design issues are addressed early in the development lifecycle, problems are minimized and the migration from test to production is more easily accomplished.

If you haven’t read the design review blog, you should. I’m intentionally not covering the importance of rigorous testing of any change before it is implemented in production because it is covered in-depth in the design review blog. From simple changes to new application implementations, there is simply no reason not to perform the test, review, change, test, review, change iteration lifecycle. Although the blog covers new application implementations, the blog will show you how important I think it is to follow a rigorous test plan.

Proceduralize the Change Request Process
Database administrators usually support different business units with each unit having their own set of unique procedural requirements. Formalizing and documenting the change request process minimizes the potential for miscommunication between the business units, application development areas and the database administration unit. This is even more important to remote database services providers.

The notification lead-time required for the database administration team to perform a requested change should be documented and distributed to business and application development units. This will prevent your team for getting a request to migrate a database from test to production in the morning with a required date for that afternoon. Of course, we all know that never happens.

If your organization doesn’t have a formal change request process in place, (and many shops don’t) create your own! There are dozens of change management and source code versioning tools available on the market today. The prices can range from thousands to tens of thousands of dollars.

Although I highly recommend these types of products, I wouldn’t let the lack of having one prevent me from formalizing the change management process. You do the best with what you have.

OK, so you don’t have the good fortune of having a formal change management process in place. What do you do?

Althouth our internal practices here at Remote DBA Experts differ because of the advanced communication and ticketing products we offer to our customers, I’ve come up with a set of “time tested” best practices and procedures that can be used by any shop. Since I have over 25 years of experience with Oracle, that’s a lot of time to test!

You can begin the formalization of the change request process by:

• Creating standardized change request documents
• Establishing change management meetings
• Creating Service Level Agreements (SLAs), which include change request turnaround times

Standardized Change Request Documents
Standardized request documents help to increase the quality of the change request process. The forms are sent to the database administration unit by the application owner of the data to be processed. Any requests not sent or signed off by the application owner should be rejected. Keep copies of all completed work requests for auditing purposes. Application owners can be virtually any member of the organization that is identified as having the authority to sign off on change requests. The most common persons are application development team leaders, section heads, department heads, etc..

Each request form contains the following common information:

• Form identifier – naming convention that allows the form to be easily identified
• Application name
• Database name
• Name and contact information of the person requesting the change
• Request date
• Required date
• Application owner signoff
• Data security signoff (if required by shop standards)
• A free form request area for nonstandard requests
• An area that the DBA will fill out when executing the change that contains the following information:
  • DBA executing change
  • DBA contact information
  • Date and time change was processed
  • Verification procedures followed (very important)

Here are a few examples of specific forms that will help formalize the change request process:

Oracle Authorization Request Form
This form is used for requesting authorization changes to the Oracle database environment. Here’s an example of one of the earliest Oracle authorization forms I created over 20 years ago. This would still work!

The Oracle Authorization Request Form contains the following information pertinent to Oracle authorization requests:

• Grantee listing for security grant or revoke
• Type of security granted or revoked

Oracle Test Environment Change Request Form
The Oracle Test Environment Change Request Form will be used for requesting physical changes to Oracle test databases. It is used for requesting both new objects (tables, indexes, views, stored programs, etc.) and object alterations (adding new columns, changing datatypes, adding columns to indexes, etc.). In addition, the form notifies the DBA team to perform ad-hoc activities (test database refreshes, exports, etc..) in test environments.

The Oracle Test Environment Change Request Form contains the following information pertinent to Oracle change requests:

• Schema owner of object to be changed or created
• Object name to be changed or created
• Object type (i.e. table, index, view) of the object to be changed or created
• Detailed description of change requested
• Data administration sign off for new data objects
• Free from area for ad-hoc activities

Oracle Production Environment Change Request Form
This form will be used for requesting the migration of Oracle objects (databases, tablespaces, tables, indexes, etc.) from test to production and the alteration of existing production objects. For new database implementations, please see my blog titled “Database Design Reviews”. In addition, the form notifies the DBA team to perform ad-hoc activities in production environments.

Each Production Environment Change Request Form must have an associated Oracle Test Environment Change Request Form counterpart. If the change wasn’t made in test, you don’t implement it in production. To facilitate this process, the identifier for the test change request form that was used to notify the DBA team to create or alter the test objects should be provided on the production change request form.

The Production Environment Change Request Form contains the following information pertinent to production environments:

• Oracle Test Database Change Request Form. Allows DBA to determine if the change was implemented in test. If no change request is found, the DBA needs to determine the reason why
• Object name of the object to be migrated or changed
• Object type for the object to be migrated or changed
• What time these activities should be performed
• The name of the database the object will be migrated from (source database)
• The name of the database the object creation/alteration will be implemented in (target database)
• A freeform area for special processing required during the migration or alteration process
• Free from area for ad-hoc activities

With all of the advanced workflow, ticketing and communication products on the market today, there are numerous options available to the DBA wanting to proceduralize the notification process. Whatever product you choose, its the information that you capture and the change management workflow process you create that is important.

Change Management Meetings
If you read my earlier blog on database design review meetings, you know I’m a proponent of constant communication between all units that are involved in the change management process. Here at Remote Database Experts we make sure we attend as many of our customers change management meetings as we can. If its a database change, we want to be there to discuss it.

How often should you hold these change management meetings? As often as you implement objects in production. If your organization makes changes to production environments on a daily basis, the meetings should be held daily. This is not as big of an imposition on your time as you may think. We provide remote database services for several very large organizations that have these change management meetings on a daily basis. The process takes about 15 to 20 minutes. Not a lot of time spent to ensure that everyone knows what is happening.

To shorten the amount of time these meetings consume and to make them as productive as possible, the following discussion items should be a standard part of the meeting’s agenda:

• Application name being changed
• Date and time change will be implemented
• Change description
• Potential business impact if the changes don’t go as expected (include both units affected and how they will be affected)
• Backoff procedures
• Requestor
• Tested by

Service Level Agreements
Identifying support needs and expectations is required to provide high quality support. You probably won’t be meeting all of your customer’s expectations if you don’t know what any of them are. As stated previously, each application has it own unique set of support requirements and expectations. Service Level Agreements (SLA) help to solidify support requirements and dispel any inflated expectations a business or application development unit may have. They probably won’t be aware of your current workload and resulting work request lead times until you tell them. The DBA team lead should meet with each unit supported to establish a set of measurable Service Level Agreements that include work request lead times, support activities required and application performance and availability objectives.

Wrapup
This is by no means an all inclusive list of activities you need to perform to standardize the change request process. It is intended to give you a head start in the right direction.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Introduction
A methodology can be loosely defined as a body of best practices, processes and rules used to accomplish a given task. The task in this case is to evaluate and select information technology products. The product could be a tool used by technicians or an end-user application that solves a business problem or improves efficiencies. The success of any methodology depends on its continuous refinement. As a result, all methodology documents should be considered to be “work in progress.” The steps contained in this section are not meant to be followed verbatim. They should be considered as a set of general recommendations to be used to increase the quality of the third-party application evaluation process.

Initial Analysis
Clarifying the business needs takes place during this phase. Individual, departmental and corporate requirements are evaluated. The focus of the analysis is on solving the business or technical problem as opposed to buying a product. End users may have conflicting requirements. Needs clarification helps the users in synthesizing a common statement of the overall requirements. Analysis to determine if existing products can solve the problem is performed at this time. The make vs buy analysis is also completed during the initial analysis phase.

The following questions need to be addressed during the initial analysis:

• Can the solution be provided with existing product sets?
• Should the product be purchased or built in-house?
• Are other business or technology units able to utilize the product?
• Does the product provide any other additional benefits?
• What is the impact of not solving the business or technology problem?

Determine Impact to the Information Technology Support Infrastructure
The product’s affect on the support infrastructure is determined during this phase. Although the major focus should be on solving the problem identified in the initial analysis, it is proper to allow external considerations to affect the final outcome. A product is useless if it cannot be successfully implemented and integrated into the existing information processing environment. The product’s impact on individuals and groups as well as its impact on existing products and technologies is determined at this time.

Collect the following information during this phase of the analysis project:

• The use of new products often increases development time and training costs. Can the additional time and higher application development costs be justified by an increase in functionality or competitive advantage?
• If the product requires support from the development area, can the application personnel effectively administer the selected product?
• Risk vs Benefit. Is the additional functionality/competitive advantage worth any additional risk the product brings to the business or IT units? This is often described as the product’s comfort ratio.
• Identify changes to the organizational infrastructure required to support the product (staff additions, training, etc.)
  • Identify changes to other technologies and products.
  • Identify additional products required to support the selected product.
  • Identify changes to existing technologies and products the selected product will require.
  • Identify changes to policies and procedures the selected product will require.

Analysis Evaluation
The analysis evaluation determines the resources required to evaluate, select, implement and support the selected product. Some problems may not be cost-effectively solved by existing technologies. Evaluation personnel must estimate the costs required to evaluate the competing products. In other words, don’t spend $100,000 to evaluate a $1,000 product. As a rule of thumb, the evaluation, selection and procurement costs should range from three and ten percent of the project budget. The time required to perform the evaluation is also determined during this phase.

Perform the following activities during this phase:

• Determine and estimate the resources required to further evaluate the business needs.
• Determine and estimate the resources required to perform the product selection.
• Estimate general implementation and customization costs.
• Estimate on-going support costs, including staff requirements and training.
• Determine timelines and deliverables for the evaluation, selection and implementation processes.
• Determine date when the recommendation will no longer be valid (recommendation life-cycle).

Obtain Business Unit and IT Management Commitment
One of the major activities of any evaluation process is acquiring management commitment for the evaluation, selection and implementation stages. Representatives from user and technical management need to be a part of the evaluation process to ensure their continued support. Management always asks the same questions. As a result, it is relatively easy to prepare for them. A satisfactory answer to “What business problem are you trying to solve?” is usually followed by “How much will it cost to purchase, implement and support?”, and “What is the risk if we don’t solve the problem?”. Management wouldn’t be doing their job if they didn’t ask these questions. To be successful in this phase, you need to be prepared to answer them.

Obtain business unit and IT management commitment on the evaluation process, selection process and possible implementation by providing the following information:

• Reasons why you are performing the evaluation (What business problem are you trying to solve?)
• How the product solves the business problem.
• What you intend to test.
• Time and cost estimates for product selection, product purchase, implementation and on-going support (Is the solution worth the cost?)

Create Evaluation Team
The evaluation team is created in this phase. For products that do not have a wide impact and are relatively inexpensive, the evaluation team will consist of a few select individuals. But for products that have a wide impact or are expensive, it is prudent to involve a larger group, some with a technical background, others with a user perspective. Wide representation and diverse viewpoints obtained early in the decision process produce better decisions and fewer surprises later. It is imperative to include all those affected: IT personnel who will be supporting the product, IT groups affected by its usage and end-users. Team leaders are also identified during this phase.

The evaluation team consists of representatives from:

• Infrastructure support (personnel who are experienced in related technologies or who would be affected by the product implementation).
• Application developers who will be supporting the application.
• Business unit champions and end-users.

Locate Potential Vendors
Identifying the vendor offerings that will be evaluated takes place in this phase. The problem is usually too much information as opposed to too little. The best way to create a potential vendor list is to take a structured view of the marketplace and investigate all avenues to give your evaluation team the best possible chance of finding the alternatives.

A few sources of locating potential vendors follow:

• Internal personnel who are experienced in related technologies (evaluation team).
• Industry research groups- Gartner, Forrester, Burton, IDC.
• Internal research – trade journals and trade shows.
• Vendors that have existing relationships with your company (i.e. IBM, MSOFT, Oracle, etc.)
• Internet search engines.

Initial Elimination
This phase will eliminate products that should clearly not be considered as viable alternatives. The overall intent is to reduce the number of candidates being evaluated. With each additional candidate, the cost of the evaluation increases. The key to a successful elimination process is to create an in-depth set of evaluation criteria that will be used to evaluate the potential candidates. Using weighted measures helps to identify the measurement criteria that are most important to solving the business problem. If the product passes this phase, it is passed to the vendor evaluation phase for a more thorough examination.

Perform the following activities during this stage:

• Create the vendor selection process (methods used to perform the evaluation) and evaluation metrics (evaluation criteria used to compare the products). Document both the evaluation methods and metrics. The evaluation metrics will include:
  1. Vendor Assessment (financial stability, size, alliances, etc.). The best product doesn’t always win. The more expensive the products are, the more time you need to spend reviewing the vendors making them.
  2. Functional requirements – evaluate product based on its ability to solve the business problem.
  3. Technical requirements – evaluate product based on its technical capabilities.
  4. Create a general requirements document containing a compliance matrix that uses weighted measures to show how each vendor meets the requirements.
• Create and distribute a Request for Information (RFI) document using the evaluation metrics created above as the basis for the document.
• Evaluate the vendor responses to the Request for Information request.
• Reformulate the requirements based on vendor input.
• Create a vendor short list to keep evaluation costs to a minimum.

Vendor Evaluation
An in-depth evaluation of each of the remaining products takes place in this phase. The processes and methods created in this step are more detailed than those created in the initial elimination phase. Vendor responses to the Request for Information may also require alterations to both evaluation processes and evaluation metrics. Remember that an individual product’s feature set is usually a compromise of function, speed and cost when compared to its competitors. You need to determine which factors are most important to you and weight them more heavily during your evaluation. A Request For Proposal (RFP) helps to formalize the evaluation process. The Request For Proposal details the scope of the sales process and defines the procedures and rules to be followed by the vendors. Vendors are more enthusiastic about committing resources to the sales process when it is clear to them how they are to respond and how their responses will be evaluated.

Perform the following steps in the Vendor Evaluation phase:

• Create a final vendor selection process (methods used to perform the evaluation) and weighted evaluation metrics (evaluation criteria used to compare the products).
• The evaluation processes and metrics created in this phase are more in-depth than those created during the Initial Analysis phase.
• Vendor responses to the Request for Information may require alterations to both evaluation processes and evaluation metrics.
• Functional requirements – evaluate the product based on its ability to solve the business problem.
• Technical requirements – evaluate the product based on its technical capabilities.
• Create Request for Proposal (RFP)
  • The Request for Proposal is more formal than the Request for Information.
  • The document details the scope of the sales process (meetings, presentations, product demos, trial periods, timelines and deliverables).
  • It also defines the procedures and rules to be followed by vendors during the evaluation process. Ground rules are written and shared with the qualified vendors.
• Notify the vendors of how they will be evaluated (overview of compliance matrix).
• Vendor meetings and presentations are held during this phase.
• Create a questionnaire and contact the vendor supplied customer references.
• The evaluation matrix is used to measure product conformance to the business unit IT unit requirements.

Communicate Results
The results of the evaluation process are communicated during this phase. There are many communication avenues to choose from. A formal evaluation document is created and distributed. Presentations detailing the evaluation process are also given. Before you make your recommendation, be prepared to justify it. The major questions to ask yourself are “Did you choose the best product?”, “Did you keep the level of risk at an acceptable level?” and “Did you achieve these objectives at a reasonable price?”

The documentation provided to business and IT management includes:

• An executive summary.
• A detailed description of the vendor chosen.
• A financial analysis of the chosen vendor.
• The reasoning behind the vendor’s selection.
• What other products were evaluated and the reasons why they weren’t selected.
• An overview of the evaluation process.
• The metrics used as the basis for evaluation.
• An overview of the responses to the Request for Information and the Request for Proposal.
• The benchmark results of all competing products, if any.

Wrapup
I hope you enjoyed reading this blog on third-party product evaluations. As I stated in the introduction, the intent of this blog was to not influence you to follow the steps provided verbatim. This contents of this blog should be considered as a set of general recommendations that can be used to increase the quality of the third-party product evaluation process. Happy evaluating!

Thanks for Reading,

Introduction
Let me begin by stressing how strongly I feel about this topic. As I stated in my first blog of this series, I always started my Oracle backup and recovery class with this statement “The fastest way to lose your job in this profession is to lose data for your company. You can be a Tom Kyte and a Jonathan Lewis X 2, but if you can’t recover a database, you aren’t of any use to your employer.”

That always seemed to ensure that my students paid attention during the remainder of the class. The backup and recovery class was one intense week of instruction that consisted of me pounding information and best practices into my students’ collective heads. I would often stop after an important topic and bellow “DO YOU UNDERSTAND?!?!!” as loud as I could. By the end of the week, the class would immediately yell back “YES!!!” as loud as they could.

What can I tell you, that style worked for me. Trust me when I say that when my students left that class, they could all backup and recover an Oracle database. The backup and recovery classes were the ones responsible for my courses being labeled as “Foot Camp” by the Oracle student population. That was OK by me.

My first “mentor”, if you could call him that, was an ex-Marine Corps drill instructor that went into IT after he retired. When I started my career, he was the first senior DBA I worked for. Every so often, he would walk up to the back of my chair as I was facing my terminal, lean in real close to my ear and say “You know what Foot, the next time I see you make a mistake, I’m not even going to tell you. I’m just gonna wait 5 minutes then come back and kick your *&^%$ up around your shoulder blades.” Not motivational for sure, but I made few mistakes. Maybe some of that rubbed off…

Oracle Backup and Recovery
Recovering an Oracle database is a wonderfully complex task. Data files, log files, control files, full backups, hot backups, RMAN and point-in-time recoveries all combine to make many administrators lie awake nights thinking about whether their databases can be easily recovered (or not).

The next few sections will provide some useful information on the Oracle backup and recovery process. My intent is to not cover any technical topics in-depth. You can get that information from a myriad of sources. My focus will be on the non-technical tips and tricks that will help you improve your recovery skills and ensure trouble free recoveries.

It’s the Little Things That Bite You
Most botched recoveries can be attributed to human error. Make sure all tapes have proper retention periods, verify that all backups are executing correctly and run test recoveries on a regular basis. Don’t let missing tapes or backups cause you to lose data. You don’t want to hear UNIX support say “the retention on that tape was supposed to be how long?” in the middle of a recovery. COMMUNICATE with others that are responsible for all other pieces of the recovery “pie” (system admins, operators) on a regular basis to ensure you have everything you need to recover a crashed database. Pick a database, identify the backup output files and verify that they are available when you need them. Remember, YOU are the technician that is ultimately responsible for ensuring that your organization’s databases can be recovered. Not O/S support, operations, application developers….

Document Your Recovery Environment!
OK, I’m yelling already. I already covered the importance of good documentation in a previous blog. You know by now that I work for a remote database services provider. It is absolutely imperative for us to know EVERYTHING about our customer’s existing backup and recovery strategies. Part of our assimilation process is to document our customer’s environment. Here’s a quick list of some of the questions we ask. The document we actually use is a standardized Word template that uses drop downs, text boxes and help buttons but this shortened text document should provide you with a starting point to help you build your own backup and recovery documentation library.

Keep Your Skills Sharp
Don’t let your recovery skills get rusty. The more test recoveries you do the easier the production recoveries become. Create one database that you and your fellow administrators can trash on a regular basis. Take turns and make a game out of it. DBAs can be pretty creative when causing problems for others when it’s all in fun.

It can actually become quite an interesting game competing for bragging rights over who has the current title of “the most devious database destroyer.” During one of my test recoveries, I couldn’t even bring up the monitor. I waltzed down to the server room and saw an open drive bay on our test server with a bundle of unconnected wires sticking out. There was a single note attached below the opening telling me to look for the next note. 15 notes later and I found the drive. Dumped it in, fired it up and found that the database was deleted. I recovered it from a tape backup. THAT was devious.

If you are a senior-level DBA, make sure you keep the junior folks on their toes. I have never personally seen the database make a mistake during the recovery process. That leaves incomplete backups and DBA error as the most likely causes of “good recoveries gone bad.”

At Remote DBA Experts, we have dozens of customers that we have to support. We test recoveries and failovers on a regular basis.  Ensuring our recovery skills are sharp is that important to us. I still do test recoveries. Its important to me to ensure that I am ready to go when the time comes. 

RELAX and Plan Your Attack
When you are notified of a database failure, take a deep breath and relax. Don’t immediately begin to paste the database back together without a plan. Create a recovery plan, put it on paper, have others review it if you can, and then execute it. You shouldn’t be trying to determine what the next step is in the middle of the recovery process. I will plan my attack on paper for all recoveries, no matter how simple they are.

Don’t Be Afraid to Ask Others
I have over 20 years of experience using Oracle and have done my fair share of database backups and recoveries. During my career as an Oracle instructor, I have assisted in hundreds (and hundreds) of database recoveries in Oracle’s classroom environments. If possible, I still have others review my recovery strategy and recovery steps before I begin the recovery process. A second opinion may prevent you from making a mistake or overlooking a key part of the recovery process.

Don’t be afraid to ask others and don’t be afraid of calling Oracle support if you have to. That’s what they get paid by your company to do – support you. Don’t make a database unrecoverable by “guessing.” When I first took over as the Database Group Manager for a large financial organization many years ago, I viewed the execution of over 70 different commands in an alert log after a botched recovery performed by a junior DBA. An ego that was too big to allow that person to ask questions created a database that was almost unrecoverable.

The Importance of Formal Education
Read the Oracle Backup and Recovery Guides before reading third-party books. The manuals will provide you with a firm foundation of knowledge on backup and recovery strategies and procedures. Then move on to third-party books (like this one) for helpful hints and tips that may assist you in the recovery process.

Take the Oracle classes! Oracle’s instructors understand the importance of backups and recoveries. You will receive days of instruction and hours of hands-on labs. You’ll learn everything from simple O/S cold backups to RMAN incomplete recoveries using backup control files.

Oh, and now that I’m retired from teaching, you won’t have to worry about me yelling at you.

Thanks for Reading,

Chris Foot
Oracle Ace
Director of Operations 

This blog will provide you with some hints and tips that will help you obtain the most benefit when you attend “Old OU” (Oracle University). I’ll also provide some recommendations to help you pass the Oracle Certification Exams.

Introduction
Readers of this blog know that I’ve been involved in some form of Oracle Education for close to twenty years now. I started my career in Oracle Education working as a courseware creator and “instructor to the instructors” for Platinum Technologies. I had the good fortune of working with fellow blogger, Craig Mullins during my stint there. At Platinum, I was responsible for creating their entire Oracle curriculum including classes on general administration, backup/recovery, database tuning and several SQL and PL/SQL courses.

I was also employed as an Oracle Certified Instructor for a number of years. The greatest accomplishment in my career came was when I was awarded the OAEC Quality Achievement Award. These awards are given to the top instructors in the OAEC (Oracle Authorized Education Center) program. That recognition was quickly followed by Oracle allowing me to certify a few of my fellow instructors. Certifying my fellow instructors was a responsibility I took very seriously. One of the folks I certified continues to have a very prosperous career in Oracle Education. It should be interesting if I ever attend one of his classes in the future. “The student becomes the teacher” syndrome would be an interesting dynamic. I’ll try not to critique him in the middle of his class…

Grey Market Training
Gray market training is classroom education provided by any third party other than Oracle or an Oracle Approved Education Center. The course content is created by the gray market provider. Therefore, there is no guarantee that the quality and course content of these classes will provide you with the information necessary to enhance your expertise or successfully pass the Oracle Certification tests.

Students who want to take gray market training classes should carefully evaluate the vendor that provides the training as well as the course’s content. Students should ask the vendor to provide a detailed listing of the course’s curriculum as well as the background of the instructor who will be providing the training.     There are several non-Oracle education providers that you can trust.   One way to find out who the good paroviders are is to ask your fellow Oracle DBAs.    Hop on over to websites like the Oracle Community and the Oracle Technet Discussion forums and you’ll get a lot of responses to your questions on third-party education providers.   Its hard to beat Oracle training if you want to prepare for certification.

Oracle Classroom Education
When is the best time to take the classes? This may sound trite, but it is best to follow Oracle’s recommendations on the sequence of classes. Take the intro classes before taking the more advanced classes. If you have the luxury (meaning you aren’t the only DBA in your shop), gain some day-to-day experience before taking the more advanced classes (SQL or database tuning, backup and recovery, etc.). You shouldn’t be asking questions like “What is an init.ora parameter file, anyway?” in an advanced class. Instructors don’t have the time and your fellow students won’t have the patience to bring you up to speed before continuing on to more advanced topics.

Preparation
Being prepared is the single most important factor to obtain the most benefit from any formal training class. I can’t stress this highly enough – PREPARATION IS THE KEY TO SUCCESS.

It’s a pretty simple process to prepare for class. Oracle Education’s Website will provide you with all of the information you need to prepare for your upcoming class.

Step one is to read the information provided by the Oracle Corporation that pertains to the class you are attending. Sounds simple enough, right? That’s because it is. Oracle Education’s website provides a detailed description of each class it offers. It will tell you EXACTLY what to prepare for. Go to the Course Curriculum page on Oracle Education’s website, find the course you are attending, navigate to the course description page and then review the course objectives and topics.  You will find a detailed listing of all of the information the course will cover.

If you are a new Oracle DBA and will be attending your first database administration class, read the Oracle Concepts Manual, Administrator’s Guide and review the Reference and SQL Reference books. Follow this same advice for whatever class you are taking. If it’s a new features class, read the new features guide for that release. Tuning and performance – read the Performance Tuning Guide. Same thing for backup and recovery, standby, RAC, etc.. Do yourself a favor and perform some self-education first. You may not know the mechanics, but you’ll be ahead of the game if you know the lingo and the concepts used.  You will pay a lot for these classes, you need to make every effort to take advantage of the information the instructor is going to convey to you.   You do that by preparing for the class.

Showing Enthusiasm and Willingness to Learn
If an instructor sees that you are making an extra effort to overcome your lack of day-to-day experience by coming in early, staying late and being prepared, they will be more prone to help you.

It’s basic human dynamics. If the instructor walks by your work area and sees you continuously reading “NFL Today” when you should be doing labs, they are going to spend less time with you and more time on students who are exhibiting a willingness and enthusiasm to learn.  Except when I was teaching, I would just stop in my tracks and ask you if you wanted to learn your chose profession  (or not).

I personally liked to keep my classes lively.  Every so often I would jump out of my seat and yell “DO YOU UNDERSTAND!”.   By the second day people would be yelling “YES!”.   Of course by Friday some of my students were looking at me like  “If you say the word SGA one more time, I will come up there and jam your head into the projector…”  I wanted to make sure I didn’t miss anything so breaks were always at a minimum.  I think that is why my classes were nicknamed “Foot Camp” by my students.

Instructors are weird; we like to see people excited about what we are teaching. Seeing someone enthused about learning makes us want to make sure they get the most out of class.

Class Participation
Class participation means more than just answering questions that are asked. You also need to ask questions. If you still don’t understand, ask again. If you are having a hard time understanding a particular concept, ask the instructor during a break to explain the information to you again.

Trust me, your instructor will be glad to help. Instructors don’t choose training as a profession because they like living out of a suitcase; education is a passion for them. I knew it was time for me to retire when I woke up one morning in a hotel room, looked out the window and couldn’t remember what city I was in. After I cleared my head (and looked at the local phone book), I found out where I was and started contemplating that it may be time for a career change. Coming in my front door that very same weekend and having my German Shepherd attack me because he forgot who I was pretty much confirmed my decision.

If you are experienced DBA and you have a lab partner that isn’t as well versed in Oracle as you are, take the time and help them. Don’t take control of the keyboard and blast through the labs. Be patient with them, show them the little tricks of the trade you picked up and help them along. You’ll probably have their undying gratitude and you’ll be contributing to the health and welfare of our profession.  You also learn by teaching.

If you are an Oracle expert taking one of the more advanced classes, don’t let your ego get in the way of you obtaining the utmost benefit from the class you are taking – ask questions and get involved!

Over the years, I found that the discussions that were held during class breaks were often just as illuminating as the instructor led training. One DBA would ask a question about a problem that occurred or the challenges they were facing. Their fellow DBAs would immediately provide recommendations to help them. Don’t be afraid to ask your fellow DBAs and instructor for advice. If you have a problem, an Oracle class is the perfect forum to obtain the information you need to get it resolved.

Familiarize Yourself with the Next Day’s Material
I know this can be hard to do. Especially if you are attending class in a place you’ve never been to before. It’s a tough decision between sightseeing in San Francisco or planting your nose in an Oracle Admin I class guide. You’ll need to make some compromises here. Go out for a few hours, come in early and prepare for the next day’s class.

The Importance of Becoming Certified
Becoming certified is an essential ingredient for all Oracle technicians who want to prosper in their chosen profession. Certifications are especially important to those seeking employment in a field that often has many candidates competing for a single position. An Oracle certification shows the potential employer that the candidate has made the commitment to “learn their trade” and has the background and education to quickly become a productive member of the their staff.

The Oracle Certification also gives a distinct advantage to DBAs looking to advance in their careers. Being an Oracle Certified Professional raises their visibility in the organization, which is a requirement for career advancement in most shops.

Preparing for the Oracle Certified Professional Exams
The best time to take the exam is a week or two after taking the Oracle class that the exam pertains to. Passing the OCP test is much easier when the information is fresh. The class workbook should be used as the primary study guide. I have passed every exam I have taken by studying only the information contained in the class workbooks.

Practice tests will provide you with a firm understanding of the areas you are strong in as well as the areas where you need to shore up your knowledge. Oracle University has chosen two partners (Self Test Software and Transcender) to provide authorized Oracle Certified Practice Exams.

I am fully aware that there are many competing websites that offer Oracle Certification practice tests. If you want to prepare for an Oracle Exam by purchasing a practice test, why not purchase the test from the vendors that Oracle has chosen to partner with?

Self Test Software and Transcender practice tests provide a thorough coverage of the Oracle certification requirements and use the same test question technology as the real exams including simulations, scenarios, hot spots and case studies. This makes the choice clear – go with the Oracle partner practice exams.

Oracle exams consist of single response (Y/N or T/F), multiple choice and for some exams, Interactive Items.    The person taking the test hits the START button and is then presented with a live simulation of an Oracle database environment.  They must then take the appropriate course of action in the simulation enviornment to earn a correct answer for that simulation.    If you aren’t happy with your previous response, you can always hit the START button again and record a new one.

If you are unsure of a question, you can always mark the question for review and at the end of the test you will be prompted to go back and review all questions that you marked.    Remember, you need to answer all of them.

The Day of Your Test
Oracle used to partner with Prometric Testing Centers to provide testing centers throughout the United States.    That will change on September  2009.    Oracle has recently chosen Pearson VUE as their new testing provider.  All future Oracle certification exams will be provided  through the Pearson VUE Authorized Test Center Network.   Pearson VUE states that it has over 5,000 sites in 165 countries.   Here’s more information on the new relationship between Pearson VUE and Oracle.

The following hints and tips will prepare you for the day you take your certification tests:

• You must have two forms of identification, both containing your signature. One must be a government issued photo identification.
• Try to show up early (at least 15 minutes) before your scheduled exam. If you show up late, the testing center coordinator has the option of canceling your exam and asking you to reschedule your test.
• You cannot bring any notes or scratch paper to the testing center. Paper will be provided by the testing center and will be destroyed when you leave.
• Testing center personnel will provide you with a brief overview of the testing process. The computer will have a demo that will show you how to answer and review test questions.
• Don’t leave any questions unanswered. All test questions left unanswered will be marked as incorrect.
• Your exam score is provided to you immediately and the exam results are forwarded to Oracle Certification Program management. Make sure you keep a copy of your test results for your records.
• If you fail a test, you must wait at least 14 days before retaking a proctored test.  There is no retake waiting time for non-proctored tests.  A proctored test means you show up at the testing center and a person watches you take it.   So don’t write the answers on your sleeve!
• Don’t let your ego prevent you from studying. I have seen numerous experienced DBAs receive “less than stellar” test scores because they thought they knew the material they were being tested on.  STUDY – The more prepared you feel you are for anthing, the less anxious you will be.   Hey, how hard can they be?  I passed every one I took!  I’m probably average on the intelligence scale (no cracks please).   Studying the classroom guides did it for me.

Wrapup
I hope you enjoyed this blog on Oracle Education.   If you have any questions on Oracle Education or the certification exams let me know.

Thanks for Reading,

Chris Foot
Oracle Ace
Director of Operations

The Perils of our Profession
One of the common challenges that all DBAs face, no matter what vendor’s database they work on, is the absolute attention to detail our profession demands. Switch a couple of characters in a script, forget to set your SID, set the wrong flag at the wrong time and the end result usually isn’t very pretty. Many commands we issue on a regular basis are destructive by their very nature. This is the reason why I have a great respect for all technicians who have selected database administration as their chosen profession.

I know they have all experienced that uncontrolled “eye-twitching” at 2 AM when they are ready to hit the final enter key to execute the command. You know what command I’m talking about too. It’s that one command that you really, really, really hope is going to come back with a successful return code and ultimately end with a database that is finally usable. Whether it’s a recovery, a file fix or corrupt data is immaterial, it’s the wait that we are talking about.

There is no longer wait in our profession than waiting for the message below after a database recovery:

SQL> Database opened.

Time always seems to stand still. The longer the recovery, the messier the recovery.  The more critical the database – the longer you wait. You stare at the screen hoping beyond hope that the above message will appear. It’s the ritual cross your fingers, spin around three times, face towards Oracle headquarters and pray to everything that is Larry Ellison wait. I’ve actually caught myself mumbling, “Come on, come on, come on…” I don’t care how sure you are of your capabilities, or how much of an Oracle “Ace” you are – you know the anticipation I’m talking about.

You then either breathe a sigh of relief or you are in absolute disgust when you see an Oracle error message appear. How about the old “File 1 needs more recovery to be consistent” or the “File 2 not restored from a sufficiently old backup”? Those messages are enough to make anyone cringe. I’m an ex-Oracle instructor. I’ve seen those messages A LOT in class. I still cringe.

At a previous job, I once had to run through 36 hours of tapes to restore a multi-terabyte warehouse. A disaster occurred that required us to do a recovery. THAT was the longest wait for a database open message I ever experienced. One of my fellow DBAs asked if I needed a brown paper bag to breathe into.

Or it’s the command that drops the schema in the test environment that will allow you to do a refresh from production. It’s that test database that runs on the same box as production. The environment that makes you do a “SELECT name FROM V$DATABASE” command in SQL*PLUS about 15 times in a row before you finally execute the “DROP USER CASCADE” statement.

Not only must we try to prevent our own mistakes, we must safeguard our environments against the mistakes of others. Operating system administrators, disk storage technicians and application developers are just like us. We are all part of the human community that makes mistakes from time to time.

If you never make mistakes, send me a resume. I’m always looking for a “Patron Saint of Oracle” here at Remote DBA Experts. It will also save us on travel costs because I’m sure you’ll be able to spread your wings and fly here on your own.

But as my old boss Dan Pizzica used to tell me (when I was a VERY junior DBA) “It really doesn’t make a difference who broke the database. You are the technician who is ultimately responsible for fixing it. The buck stops with you. If you can’t protect your environments, you aren’t doing your job.” We all know he’s absolutely correct.

Then there’s the software glitches. The problems that pop up out of the blue and make you go:

“WHAT THE? – How did THAT happen? I’ve done this 317 times in a row and it worked every time.”

For you math majors, here’s my calculation for this:

CLOSER YOU ARE TO PRODUCTION TURNOVER
+ THE GREATER THE VISIBILITY OF THE PROJECT
= THE MORE LIKELY A PREVIOUSLY UNKNOWN SOFTWARE GLITCH WILL OCCUR

I don’t care what software you are using, you will run into the “only occurs on this release, on this version of the operating system, using this particular feature on the third Tuesday of the sixth month when it’s cloudy outside” BUG. Be sure to expect management to stop by and ask “well, why didn’t you test this on the third Tuesday of the sixth month when it was cloudy outside?”

The more complex the database ecosystem, the more paranoid I become. Which is why I’m not a follower of “the database is getting so easy – we won’t need DBAs” mantra that mindless industry pundits profess on a seemingly endless basis.

So now we know that our jobs are somewhat unforgiving and we do make a mistake from time to time. What can we do to reduce the chance of an error occurring?

Poka-Yoke for DBAs!
We recently had a Poka-Yoke contest here at Remote DBA Experts.   We hold contests on a regular basis to jumpstart the creative process on activities that we feel strongly about.   We are big proponents of Poka-Yoke”.  Poka-Yoke is a Japanese term that means “fail-safing” or “mistake- proofing”.   Wikipedia’s definition of Poka-Yoke is: “its purpose is to eliminate product defects by preventing, correcting or drawing attention to human errors as they occur.”

Since I’m a car nut, here’s a couple of automotive Poka-Yoke examples.    You can’t take the keys out of most modern cars until the car is in park.  In addition, most cars won’t allow you to shift out of park until the key is in the “ON” position.   How about gas caps that have the little tether that prevents us from driving off without the cap?   Most gas caps are also attached using a ratchet assembly that ensures proper tightness and prevents over tightening.

Take a look around you, you’ll see dozens of Poka-Yokes during your daily activities:

• The little holes in bathroom sinks that prevent overflows
• Microwaves will stop when the door is opened
• Same thing with dryer doors
• Lawn movers that have a safety bar that must be depressed before they will run
• Disk brakes that begin to make a noise before they are completely ground down
• Rumble strips on roads

The list really is endless.  We have applied the Poka-Yoke process to our daily activities here at Remote DBA Experts.  We have checklists, process documentation, best practices, sign-off sheets – the works. One of the first blogs of this series is a discussion on the importance of good documentation.

I’d be very interested to learn your Poka-Yoke ideas!   If you have a Poka-Yoke idea, please respond and we’ll be glad to discuss it.  Here’s some general ones that I recommend.

The Second Set of Eyes
As I have stated in previous blogs, I have over 20 years of experience using Oracle and have done my fair share of database backups and recoveries. During my career as an Oracle instructor, I have assisted in hundreds of database recoveries in Oracle’s classroom environments. If possible, I still have others review my recovery strategy and recovery steps before I begin the recovery process. I used backup and recovery just as an example. Whatever the process is you are are performing, a second opinion may prevent you from making a mistake. A review from a fellow DBA has saved me more than once. I may be described as having an ego (I have no idea where they get that opinion) but it doesn’t prevent me from asking for help from others.

We were recently correcting a very poor backup script created by a customer’s previous database support vendor. The customer described this particular environment as “if it goes down we lose our ability to make money” application. After our massive set of changes were complete, two of us went line-by-line verifying each line of the backup script. At the end of each script we asked each other “are you OK with this”? Only then did we move on to the next one. I don’t care how much time you have “in the seat” using Oracle, you need to put your ego aside at times and have someone check your work on critical activities.

Concentration
I used to work for a shop that subscribed to “the everybody in one big room” philosophy. I guess it was supposed to allow everyone to work together as a team and become as “one with each other”. It may have achieved that purpose but it sure didn’t allow you to concentrate on your work very well. You could hear so many different conversations they had to pump in white noise. The constant ‘whhhsssssshhhssshhh” noise made me feel like I was a crewmember of the Starship Enterprise. I always wondered when Captain Picard would walk through.

Like all DBA units, our particular area was often populated with various developers and O/S technicians. Many different conversations were occurring, some that could be described as somewhat animated. The environment did not allow you to concentrate on the task at hand. We often had to go into small conference rooms to work on critical tasks.

The point I’m trying to make is that no matter what type of environment you work in; if you can concentrate OK, but if you are like me and you can’t, find a spot where you can. Block off some time, send questions to other DBAs and concentrate on the task at hand. Don’t attempt to answer questions and code a complex script at the same time. May seem obvious, but throughout my career I have personally watched numerous DBAs attempt to multitask when they are working on a critical process. It’s a recipe for a problem. Once you are done, follow rule number one and have someone review your work.

What Database Are You Working IN?
Working in the wrong database is a common problem for database experts as well as their less experienced counterparts. How many times have YOU found yourself running statements in the wrong environment? Feel free to include me in that not so select group. The operating system command SET can be used in Windows systems to display environment variables. The ENV command can be used to display the environment variables in UNIX. Many seasoned database administrators change their UNIX shell prompt in their profile to display the current Oracle SID. Displaying the current Oracle SID in the shell’s prompt provides a continuous reminder to the DBA of the database they are working in.  Google it – you’ll find dozens of scripts by your fellow DBAs.

Setting your SID Automatically During Logon
Here’s an easy recommendation – don’t do it. Many of the servers we work on contain several databases that use different database software releases. It is our standard to not set environments in logon scripts. We create shell scripts that have the database name in their title to set our SID, ORACLE_HOME, etc..

It is a little harder in Windows environments. During installation, Oracle often uses the first database created on the server as the default SID. You can override this configuration, but you’ll need to read the Oracle installation guides to find out how. The manuals will show you how to override the default database on Windows operating systems.

Saving Time VS Creating a Problem
I once watched a fellow DBA perform a rather complex set of administrative tasks to solve a problem. He was rapidly flipping back and forth between at least 15 active screens, copying and pasting and editing and copying and pasting and editing… I describe this particular activity as “Multiple Screen Syndrome”.  He also had several other screens open that were connected to other databases. He was multi-tasking to its highest degree.   Take a break, take a breath and look at what you are doing.

How about the rm -r /u0*/ora*/prod*/*/*.* command in UNIX? It’s the command that drops multiple databases in multiple directories. All in one painful swoop. How many times have you heard of a mistake caused by commands like this causing mass mayhem? When you make a mistake like this, you become immortalized in conversations for years to come. Get a few technicians together after work and ultimately the conversation will include “remember when Bob so-and-so ran that big rm -r command by mistake and wiped out the entire O/S on our production web server? You can’t tell me you haven’t heard stories like this.

My opinion as a database manager is that I would rather you take your time than showcase your multi-tasking and time saving skills. The more complex and critical the activity, the more basic you should become in your plan of attack.  Trust me when I say I won’t be impressed with your time savings “cut and paste” and wildcard expertise if I think it can even remotely be dangerous.

Safety First Mindset
I once saw a DBA log in to a database using a particular schema account. He then logged into a different database using a DBA account and dropped the schema with the same name he was logged into on the first database. I asked him why he logged into the first database using the schema account he just dropped in the second. He stated “Oracle won’t let you drop a schema that is connected. No matter what happens after this, I’m positive that I won’t drop the user in this database by mistake.” I like that Safety First mindset in a DBA.

You need to think Safety First when you are performing any particular complex or critical activity. Take the time and put one or two safeguards in place like the DBA did when he dropped the user.

Other DBAs may call you paranoid, I’ll call you an experienced DBA that would rather be safe than sorry.

Wrapup
The intent of this blog was to not provide you with a laundry list of recommendations. It was intended to help jumpstart your creative juices to think about different methods to protect yourself against problems. If you have any helpful hints, please feel free to respond to this blog with your Safety First Tips and Tricks.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

You begin creating your own rules of thumb by reading opinions from folks that you can trust. The industry experts that are both learned in their chosen profession and aren’t afraid to provide you with their opinion. Spend some dedicated time evaluating authors by reviewing others’ opinions of them.

In addition to learning from others, you need to create your own test environments and experiment. One of Oracle’s great benefits is that it runs virtually anywhere. If you aren’t sure how something works and the manuals aren’t explicit enough, build a test case and execute it. At Remote DBA Experts, we experiment with Oracle 10g and Oracle 11g daily. Especially recovery testing…. It keeps our skill sets sharp and reduces stress during the “real thing”.

How do you make your own rules of thumb? Follow Jonathan Lewis’ advice and test, experiment and learn! Afraid that performance will suffer if you place 4 columns in an index to obtain index only access? Build the index in test, have the developers run transactions that access the new index, monitor on-line transaction performance and find out. Then document the results.

Try different column combinations, multiple index usage, etc. When prototyping complex joins between several tables, build a set of indexes that favors one access path and test the queries in question. Drop the first set of indexes and create indexes that favor another access path. Build the different permutations of indexes that influence the optimizer to join a different set of tables first or allow different join types to be performed. Test the different access paths and record the performance statistics.

The above recommendation may seem to be a time consuming process, but until you learn access paths, this is the best approach to take. I have spent hundreds of hours working with access paths and if I could impart one bit of wisdom, it would be the following one: No matter how much time you spend reading others’ opinions on how the database works, you need to spend time “in the seat” experimenting and learning on your own.

The same advice goes for learning GRID, backups and recoveries, advanced database features and virtually every other facet of the database. You need to do as all the other experts do – test, experiment and learn. You’ve chosen database administration as a profession, not a job. When I interview candidates for DBA jobs here at Remote DBA Experts, that passion to learn and experiment plays a big part in my decision-making process. If I hear a candidate say “I have an Oracle database lab installed at my home”, they have earned themselves some bonus points with me.

The Ever Changing Database Ecosystem
One of this blog’s readers, Michael Rife, also provides an excellent recommendation to revalidate your “Rules of Thumb” on a regular basis.  Database, O/S and hardware vendors must continuously tune, tweak and generally improve their offerings to remain competitive.   As a result, the environments we support are constantly changing.   We all know that each new release of the database requires a thorough evaluation to determine how to best exploit the new features it contains.  But we must also be wary of any changes to existing features that would compel us to revalidate our own administration and implementation “Rules of Thumb”.

Repeatable Processes
Repetition, even though it can be boring, is the foundation for a high quality support environment. If the scripts and administrative process worked correctly the first time, chances are they will continue to work correctly in the future.

Automating and documenting complex administrative processes such as production to decision support database refreshes and application upgrade activities will allow future iterations of these activities to be executed more quickly and with less errors. As you continue reading my blogs, you’ll understand the importance I place on documentation. Here at Remote DBA Experts, we have built our entire foundation of customer support on documentation and database support best practices.

As I have stated in the past, have you ever tried to refresh an ERP application test environment from production when that test environment didn’t have enough space to hold all of production’s data? 4,000 steps later and you begin to second-guess your choice of professions. The more complex the process is, the greater the need for detailed documentation.

The moral of this story is: If you don’t want to be the only one that can perform that 900 step ERP Application production to test refresh, script it and then document it.

DBA Report Cards and the 360-Degree Review Process
Effective measurements are required to judge the success of any activity. The quality of support the DBA team provides should be reviewed on a regular basis. A DBA report card allows business and application development units to provide feedback on DBA support activities. As a remote services provider, we are judged daily on our ability to meet our service level agreements. Our customers are a tough bunch and we are OK with that. They have entrusted their most valuable corporate data assets to us. A responsibility we do NOT take lightly. But a report card can also provide benefits to internal DBA units. You will never know how good a job your team is doing until you ask.

The report card will allow your customers to measure how well they feel you are meeting your Service Level Agreements. Here are a few sample questions to start you on your way:

• How would you rate the turnaround times for DBA unit work requests?
• How would you evaluate the DBA unit’s responsiveness to questions?
• How would you evaluate the DBA unit’s responsiveness to requests for assistance?
• Please rank the quality of communications the DBA unit provides.
• Please rank the overall quality of work the DBA unit provides.

All questions are ranked from 1 to 5 with 5 being the highest. You can also allow the respondents to rank the importance of each question from 1 to 5 with 5 being the highest.

A short survey should be attached to the DBA report card to gather additional information that can be used to improve the quality of support the DBA unit provides to their customers:

• What are your top three technical challenges that you face?
• What are the top three non-technical challenges?
• Please list your current priorities. Rank them in order of importance.
• List the most important services the DBA unit provides. Rank them in order of importance.
• What support services does the DBA unit do a good job of providing?
• What support services should the DBA unit improve upon?
• What additional services would you like the DBA unit to provide?

Meetings can be held with the respondents to discuss their reviews. DBA team members participating in the reviews must be prepared to respond to criticism in a professional manner. But just as its title describes, the 360-degree review process also allows support units to provide feedback on their customer’s support requests and work activities. The 360-degree review process provides important feedback to both support units and their customers. Once again, your customers may not know that some of their expectations are unachievable until you tell them the reasons why.

Corrective Action Reports
Oracle is a challenging database to administer. As much as we would like to prevent mistakes, we do make them. That’s one of the benefits of being human; we learn from our mistakes. During my 20 years in this field, I’ve seen all kinds of mistakes made – from little ones to catastrophic.

The worst one I ever experienced is when we brought a technician in from a third-party disk vendor to format what we thought was to be a single disk in a huge disk array. We just purchased the arrays and were in the process of assuming support responsibilities from the vendor. The vendor assisted us in the initial setup and data migration from our old storage devices to their new whiz-bang storage system.

The array stored data from dozens of Oracle databases. After 20 minutes of showing our folks how to set the necessary switches and enter the appropriate values at the prompts, the technician hit the enter key with much flair and bravado.

I assumed we would see the one activity light activate for the disk being formatted. Instead, I saw a whole wall of lights begin flashing and twinkling. I thought to myself “Hmmm, this can’t be good…”. I glanced at the technician and the look of horror on his face confirmed my suspicions. He then reinforced my conclusions when he stammered, “I think I just formatted the entire array.” I looked at the DBA next to me and said “Looks like we are in for a looong night.”

I continue to be amazed at the rapid improvements in hardware redundancy and reliability. Hardware platforms now have the ability to recognize individual component failures, bypass them, produce diagnostic information and “phone home” to report the problem to the manufacturer. But hardware components will fail. Just as the sun comes up every morning, hardware components will fail, operating systems will get tied up in knots and database bugs will manifest themselves. Such is life for a computer professional.

The key is to document everything that happened during the time period when the problem occurred. Documenting provides you with the information you need to prevent the problem from occurring again and your customers with information on exactly what happened. Its my experience that the more informed a customer is about the problem and the steps your unit is taking to prevent its reoccurrence, the better they feel.

The corrective action report should contain:

• A detailed description of the error that caused the problem.
• The customer impact the problem caused.
• A timeline of the activities that were performed.
• The steps that were performed to correct the problem.
• Mitigating factors that contributed to or exacerbated the problem’s impact.
• The steps that will be taken to prevent the problem from occurring again. Include who is responsible for the activity and a date the activity will be completed.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery