Expedient offers a variety of managed services including virtualization, cloud computing, remote backups, management of equipment, storage area networks, and disaster recovery.  Since RDBAE realizes that our customers put a high level of trust in us, we closely reviewed Expedient’s SSAE’s control objectives for physical security, data protection, change management, internal server monitoring, and environmental devices designed to protect our customer’s servers.   Expedient had no exceptions identified on its entire SSAE 16 (Type 2).

Benefits of the Remote DBA Experts/Expedient Partnership:

• Eliminates up-front capital expenditures.  Customers are not required to provide all of the infrastructure components required to ensure high availability, reliability and redundancy
• Significantly reduces ongoing operational expenses
• Allows customers to focus on their business and not datacenter/infrastructure support
• Environments can be custom tailored to meet each customer’s unique needs
• Customer is able to pay one single bill for their entire application environment that includes both the hosting component and support services
• Support for both virtual and physical server platforms
• Customer retains the ability to provide their own OS support or select from a full suite of remote OS services
• Utility computing model allows customers to quickly scale capacity to meet business needs
• Hosted environment can be configured with high availability and disaster recovery features to meet customer’s availability requirements
• Industry leading Service Levels
• Quick deployment feature reduces time-to-pilot and test projects
• Microsoft SQL Server database  and OS software can be included in the package

It is important for us to work closely with a company that offers a superior level of service that our customers expect from a hosting provider, and we are looking forward to a rewarding partnership with Expedient Communications.  

The intent of this article is not to have the database detective ask each and every question verbatim, but to take a structured approach to the information collection process.     The purpose of these questions is to define the problem and narrow the scope of the issue.   Narrowing the scope is the key to quick performance problem resolution.   The DBA can then focus on a specific issue rather than taking a shotgun approach to performance problem analysis.

It assumes the DBA knows very little about the application.   It is obvious that DBAs who have supported any application for a length of time will inherently know the answer to many of the upcoming questions.

Application Description
These questions assume the DBA is new to the environment (a consultant would be a good example).  Their intent is to allow the investigator to have a better understanding of the application that is experiencing performance issues.

• What is the nature of the application?  What does it do?
• Is the application a third-party product or internally built?
• If the application is from a third-party, who is the vendor?
• How many users does the application support?
• Does the application permit users to access the data using AD-HOC query tools?
• Does the application have any batch processing work executed?
• Are users allowed to schedule AD-HOC batch work in the system?

Identifying the Problem
The investigation continues by attempting to narrow the scope of the problem.  It is obvious that the answers to one question may answer several others.    This will allow the DBA to limit their upcoming technical investigation to the specific problem.

• What is the nature of the problem?  (i.e. entire application slow, group of transactions, specific transaction, individual screen)
• Are all users experiencing the problem or is it localized to a single user, group of users or a geographical business site?
• What is the user’s evaluation of the performance issue?  Is it 2 times as slow, 3 times as slow?
• What impact is the performance problem having on the users?
• Which performance objectives are not being met?
• Can the problem be recreated on demand?

Establishing a Timeline
Identifying when the performance issue began to occur is critical to the investigation process.    The DBA will be able to use this information to determine if any business and/or technical changes were made to the environment.    These changes are often the root cause of the performance problem (see section on identifying changes below).

It is also important to determine if the problem can be associated with a particular time of day, day of week or month.     Answers to these questions will allow the DBA to investigate possible workload contention issues (see section on workloads below).

• When did the problem start occurring?
• Has the problem existed from the time the system went into production or has it occurred after implementation?
• Does the problem occur intermittently or is performance poor all of the time?
• Does the problem occur during specific times of the month? (i.e. month-end, beginning of the month, mid-month)
• Does the problem occur during specific calendar days (i.e. Monday, Friday)?
• Is the issue more prevalent at particular times during the day?
• Is the problem getting worse?  If so, is it daily, weekly?

Identifying Changes
identifying the exact date when performance began to suffer allows the DBA to determine if anything else changed at the same time.    Many of the database performance monitoring tools allow users to review performance metrics historically, but answer to the questions below will certainly speed the investigation process.

• Did anything change from a business perspective (adding more users, increasing data volumes)?
• Did anything change from a technical perspective (software upgrades/patches, new application code changes or new code implemented, database moved to another server, adding databases to server, hardware changes)?

Are Workloads Causing the Issue?
Answers from the previous questions, that attempt to identify the particular time of day, day of week or month that the performance problem occurs, allow the DBA to draw correlations between workloads and performance.   Workload increases and workload contention are common causes of performance issues.

• Are there databases used by other applications on the server in question?
• Are there any application batch jobs running when the problems occur?
• Are there any system jobs running when the problems occur (i.e. maintenance jobs, etc)?
• Is there an increase in the number of online users during the times when performance is an issue? Are more people using the system at that time?

Application Technology
As stated previously, if the DBA has supported the application for any length of time, this information should be obvious.  If the entire application is slow, information on the architecture and individual server environment becomes very important to the investigation.    Proper sizing of the various components is a requirement for good application performance.

• Identify the application server configurations (number of app servers, web servers,  load balancers and  hardware configuration of each including CPUs, memory disk subsystems)
• Identify the database ecosystem (hardware, operating system, database):
  • Hardware vendor, model and configuration (CPUs, memory)
  • Disk subsystem (local, SAN, NAS, storage array vendor)
  • Operating system version and patches
  • Database version and patches
  • Highly available configuration

Conclusion

In my next article, we’ll use the information provided by these questions to focus our technical investigation.  We’ll run through some common scenarios and review some of the methods used to further narrow the scope of the analysis process.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Customer Value & Service Delivery

Introduction
Remember the old days? When the applications our customers used were simple and problem determination was easy? The days of true client server application architectures that consisted of a fat client application running on the PC, a network and a database. High-level performance problem determination was a breeze. It was either the PC, the network or the database.

Well, we all know those days are looong gone. We can also agree that application complexity will continue to grow and, as a result, continue to complicate the problem determination process. From Service Oriented Architectures and BPEL to advanced server configurations (and everything in-between), our environments are growing more challenging by the moment.

Performance Problem Analysis – System Triage
One of the common mistakes that is often made during a tuning exercise is not gathering enough high level information during the initial stages of the problem determination process. Database detectives get an inkling of what they think the problem is and jump right in. As I stated in my introduction, the key to success is to begin the analysis at a high level and then continue by narrowing the scope of information collection until the problem is identified.

Once the problem is identified, we tune and tweak, record the same statistics we used to identify the problem and compare the before and after information. This iteration of recording/changing/recording comparing is done again (and again) until the problem is corrected.

Asking the Right Questions to Help Identify the Problem
OK, so you have been called about a performance problem. What do you do? Suggestion – blame the network! OK, I’m kidding. What you need to do is contact the parties affected by the problem and perform a little detective work.

Before we continue, let’s take a quick look at the application architecture diagram for the sample application that we’ll be using in this discussion. It is a simple multi-tier application that uses a lightweight application on the PC, a workload balancer, one or more web/application servers and finally, an Oracle database back end. Denver and Atlanta are remote sites while the application hardware, additional business users and you are located in Dallas. Like most applications, each site runs a common set of programs with a few programs that are site specific. We also can assume that most load balancers are configured to transfer connections to the least utilized web/application server.

It’s 2:30 PM on Friday. You are counting down to the weekend. You’re thinking to yourself “this week wasn’t so bad” when the phone rings. It’s Mike the applications manager telling you that users are complaining about poor database performance and you need to fix it. The first question that comes to your mind is “Why does everyone always assume that it is a database problem?”

The DBA Detective
You decide to begin your detective work by going straight to the end-user (or users) experiencing the problem.

You ask Mike the manager for the end-user’s contact information. He gives you the number for the business team lead in Denver. You make the call and begin your interrogation. Your first set of questions are as follows:

How many users are affected by the problem?   Are they all running the same programs?   Is performance bad all the time?    When did it start happening?

If it is all users at a specific site:
You continue your investigation by calling the other remote site to determine if they are also affected by the problem.   Ask them to run the same application components if possible.   Then ask them the same aforementioned questions.

If the other remote site is not experiencing the problem, continue your own investigation but you should also place an emergency call to your friendly network specialists and ask them for help. Explain to them that you have one remote site that is running fine, while the other site is affected by poor application performance.

If the other remote site is also experiencing the problem, call one of your favorite users in Dallas. Since the hardware for the application is also in Dallas, this is an easy way to determine if the problem is localized to just the remote sites. Ask the user in Dallas if they can run the application. If the Dallas users are running fine, the performance of the remote network is now a suspect. Once again, continue your own investigation but also ask assistance from network support.

If all sites are affected:
The list of possible suspects now grows. It could be any piece of the puzzle – network, load balancing, application server, database, hardware server. One of the tricks I have used to assist in my analysis of performance sensitive applications is to talk with the application developers and ask them to send me a half dozen to a dozen performance sensitive SQL statements from their application that don’t perform data updates.

I will try and mimic the application as closely as possible. If the application is using stored procedures, it’s easy, I’ll create my own stored procedure. If the application uses bind variables, I’ll code my program to use bind variables. Here’s an example of how to use inline input to SQL bind variables that will be executed in SQL*PLUS. Please note that executing a SQL statement in SQL*PLUS requires that you define both input and output variables.

I’ll run the statement when performance is good with timing on in SQL*PLUS to get a baseline. When I’m troubleshooting a performance problem, I’ll execute the SQL statements and compare their performance against my baseline. I like to run the statements twice, once from my PC and another execution on the database server itself. If it runs fine on the database server yet runs poorly from my PC, the network is once again one of my prime suspects.  I’m also a big fan of the historical ”slider” in GRID.   It allows you to go “back in time” to analyze various resource utilizations.  More on that later.

I’ll l continue my own quick database ecosystem (database, operating system, hardware server) triage before contacting other units. I’ll be looking at CPU, memory and disk utilization and identifying processes that are consuming large amounts of resources. I’ll log on to the database itself and look at top SQL, session waits, locking and system event statistics for more information. If the database is 10G/11G, I’ll use an entirely different set of tools than I would with earlier releases.

If the statements run close to the baseline from both my PC and on the database server and my initial review of the database ecosystem looks like it is working as it should, I don’t immediately rule it out as the cause. But it will affect the way I continue my analysis. Before I begin my own more detailed database ecosystem performance analysis, I will contact the technicians responsible for the application and network tiers. All groups involved can work in unison to identify the problem.

I’ll ask the folks responsible for the application tier to log on to their web/application servers and review the same CPU, memory and disk utilization statistics that I would on a database server. The application tier folks can also review connection input/output queues. I will also ask them to compare performance statistics amongst all hardware components of the web/application tier. If one web/application server is at 100% utilization while the others are at 20%, the workload balancer now becomes one of the suspects.

I will also ask the network and server specialists to review network performance of all hops from the database server to the remote and local sites. As I stated previously, I don’t arbitrarily rule out the database ecosystem if the SQL statement runs fine on the server, but I know that other components of the application also need to be investigated.

If it is a select group of users having a problem:
If Mike the Manager states that only a subset of users is experiencing a problem, you will need to identify the specific transactions they are executing. It’s time to talk one-on-one with a business user that is having the problem. If only one user is experiencing the problem, ask them if they can log on to the application using a fellow user’s computer and execute the same transaction. If performance is acceptable, ask the user to run other transactions. If they continue to run with acceptable performance, ask them to log on to their original PC and run an additional workload. If performance is once again poor, their PC and/or network connection could be the culprits.

If it is a group of users, or the user continues to experience poor performance on all PCs they try, it’s time to get the developers involved and ask a few more questions:

• Is it the entire application running slow or just specific screens/transactions?
• How much slower is it?
• Is the problem intermittent or continuous?
• Has this problem occurred before?
• If so, is it getting worse?
• Does the problem occur at the same time every day?

In my next blog, I will provide an extensive list of questions that can be used to further narrow your analysis.  After the line of questioning is complete, you will need to determine exactly what transaction is causing the problem. It’s a relatively simple process if all users come in with unique accounts and/or program identifiers. Log on to the database, ask the user to log on and look at the sessions or top sessions screens. You can utilize GRID Control, or basic SQL statements to drill down and identify what SQL statements a specific transaction is executing. In addition, GRID also provide screens that show database locking information.

An application that uses a common ID and program complicates the problem identification process. Having an application developer on hand that knows the SQL being executed by the various transactions can be invaluable. Oracle 10G and Oracle11G provide components that allow you to monitor the current SQL being executed. More on that later…

As I stated previously, I don’t’ want to get too in-depth in this blog. In upcoming blogs, we’ll take a look at what tools you can use to identify poor performance when you find out that it is your database causing the problem.

The intent of this blog was not to persuade you to follow the steps it provides verbatim, it was to provide you with a few high-level recommendations to help you quickly determine what component is causing the problem. Look for more detailed information in upcoming blogs.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Customer Value & Service Delivery

Because of these varied roles, I have had the good fortune of meeting people in virtually all organizational roles in an IT shop.    Network Engineers, Application Developers,  OS Administrators, High Tech IT Scientists, big time tech investors, Jr DBAs, Disk Administrators, CEOs, CIOs, CTOs, CFOs and every other three letter acronym you can think of.

What I’ve found amusing over the years is how superstitious IT people are as a class of professionals.  From CIOs of Fortune 100 companies to Operating System Administrators working in single database shops, I have noticed that no one wants to talk about how good their server is performing, how great of an asset an employee is to them, how the uptime of their most critical app is exceeding their expectations – all because they are afraid that something bad will happen if they mention it.

In a previous job, I participated in a meeting at a Fortune 50 company that was attended by the “who’s who” of my organization.    I watched our CIO, a very reserved person, whose presence commanded attention from everyone involved, knock on his desk stating “knock on wood” each time he mentioned to us how an application was performing.   I counted 7 “knocks on wood” in a 45 minute conversation.

During a conference call with the CEO of another fairly large organization, I was surprised when he cut off his CTO mid-sentence when the CTO was discussing their critical server’s uptime stating that “every time you say that, something bad happens.”

Have you ever talked about how great an application’s availability was and have it “go down the tubes” that night?   Discussed how well a particular third-party product was performing with a colleague and found that performance was unacceptable the next day?      How many times have you had a co-worker state “let’s not bring that up because we all know what happens when we talk about it.”

We all know that most of this can be attributed to coincidence.   But how many times have you seen this happen?  Has it made you superstitious?     How many times have you stated “knock on wood”?

I’ve also watched technician’s little idiosyncrasies appear as they began to perform a fairly dangerous operation on <insert server, database, firewall, application, operating system here>.  Much like a baseball player who crosses themselves, knocks the dirt off his left shoe, then right shoe, every time they go up to bat, the technician performs a ritual because “it always makes it work right.”    I’ve walked into offices that range the spectrum of the IT profession and found lucky charms that the owner says “brings me and my systems good luck.”    I thought to myself “OK, that’s interesting, but I’d rather trust my own skill sets and best practices over a fuzzy keychain ornament”.   Although I refuse to think that superstition plays any role in the availability, performance and usability of any application, I have “knocked on wood” once or twice in my career.   How about you?

In my next blog, we’ll get technical and begin a discussion on problem determination techniques.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Customer Value & Service Delivery

The AWS Solution Providers Program allows organizations in the program to enhance and increase their cloud computing offerings. By developing a relationship with AWS, Remote DBA Experts can now help current customers set up applications on the cloud more quickly and successfully as well as reduce delivery times and reach a wider customer base.  As we continue to grow, our involvement in the program is sure to be beneficial to our organization.

In addition to being a member of the AWS Solution Providers program, Remote DBA Experts is now listed in the AWS Solution Providers Directory. To see Remote DBA Experts’ listing, please click here. viagra

The following is a quick run down of our stats for the event. We had 108 people signup, 85 attendees, 11 speakers covering 18 sessions on Saturday. Many thanks goes out to the speakers, volunteers, sponsors and attendees for making this a successful SQL Saturday.

The Good

• We had a great mix of MVP’s, experienced speakers and first time speakers. One of the greatest accomplishments for this event is giving Matt Velic (Check out his #sqlsat80 photo’s), Abi Chapagai and Shannon Lowder their first opportunity to speak at a SQL Saturday. The speakers all did a fantastic job. We received several comments from attendees thanking us for having fantastic speakers.
• Registration was quick and smooth. Everyone was able to get in and get to the main room to socialize and enjoy a free breakfast. The volunteers did a fantastic job.
• We took a step outside of the box with lunch and got burrito’s from Salsa Café. With only a few complaints, this was a hit. Last year we did box lunches from Panera and it was expensive and we also got lukewarm reviews.
• Speakers received their evaluations shortly after their sessions. Personally, I have spoken at a few events and never received my feedback so I completely understand how important it can be towards growing as a speaker.
• WVNCC was a fantastic host facility. The staff was very friendly and able to get us everything we needed to be successful.
• Hosted the event on a $2,400.00 budget. There will be more on this in a future blog post.

Stuff to Work on Next Year

• Speakers dinner might have been a little too early. We held our dinner at 6pm and a few speakers wanted to attend but couldn’t make it in time. We also had plans of following our dinner with a voyage of the Italian Festival but a rain storm quickly sidelined that plan.
• SQLGolf was a great idea and fun until the heat kicked our butt around the fifth hole. Once we got to the eighteenth hole I forgot we were still playing golf.
• We need more onsite sponsors next year.
• Bacon was not included in breakfast.

John Sterrett, SQL Server DBA

SQL Server File Initialization

The SQL Server storage engine utilizes an initialization process on data and log files. This process fills the files with zeros in order to overwrite any existing data that may be on disk. The initialization process occurs when you create a database, restore a database or file group, add files to an existing database or increase the size of an existing database file. These disk operations are blocking processes that will cause SQL Server to wait until they are complete.

What is Instant File Initialization?

Simply put, instant file Initialization removes the file initialization penalty by negating the need to fill the newly allocated space with zeros before it can be used. This is incredibly important and can reduce disk allocation times from minutes to milliseconds. This is incredibly important when you think about restoring a VLDB and how long you’ll have to wait just for the empty database files to be initialized.

There are a couple of caveats to mention. The first is that instant file initialization is only available on Windows Server 2003 or later. If you’re on Windows Server 2000 then you’re out of luck. The second is that this only works for database data files and is not available for transaction log allocations. The last thing to note is that this is not available if you’re using Transparent Data Encryption (TDE) in SQL Server.

Am I Using Instant File Initialization?

If you don’t know if you’re using this capability then I recommend you find out as soon as possible. Instant file initialization is only working for you if the SQL Server service account has been granted SE_MANAGE_VOLUME_NAME .

Figuring this out from the comfort of SSMS turns out to be pretty simple but it does require the use of xp_cmdshell in order for the query I’ve provided to work. If your security policies restrict the use of xp_cmdshell then you’ll have to figure this out the hard way or turn on xp_cmdshell before you execute and then turn it off after you’re done. If xp_cmdshell is enabled then execute the following script to see if you’re receiving the benefit:

EXEC xp_cmdshell ‘whoami /priv’;

GO

What have we just done? Well, we’ve just executed the Windows “whoami” command in the context of our database service account. We included the /priv parameter because otherwise this tool returns quite a bit of information and we only need to see the security privileges. It’s important to note that the output will return all the security privileges whether they are granted to the account or not. The key is to look to the right of the privilege to see whether is disabled or enabled. Have a look at the output below:

If you scan down to the bottom of the list you’ll find the SE_MANAGE_VOLUME_PRIVILEGE near the bottom. You’ll see that to the right the privilege it is listed as enabled. This is great news and we’re not paying the initialization penalty during data file allocations.

Wait, Instant File Initialization is not Enabled!

If you executed the query and came to the unfortunate conclusion that instant file initialization is not enabled, don’t worry because it’s very easy to implement.

First off, open up the “Local Policy Editor” on the database server and navigate to the “User Rights Assignment” node under the “Local Policies” node. When you select the “User Rights Assignment” node all of the policies will be displayed in the pane at the right.

You’ll need to scroll down to find policy. It’s named “Perform volume maintenance tasks.” You’ll notice that the users or groups that have been granted this privilege are listed to the right. Open the policy and add the SQL Server service account.

Unfortunately, once this is complete you’re still not finished. You’ll need to restart the SQL Server service for it to take effect.

Conclusion

I think it’s easy to see how this feature will benefit your environment in a number of situations. So, take the time today to check this out and save yourself all the waiting when time matters most.

Scott Caldwell, SQL Server DBA

You’ll find references regarding these in blogs across the Internet and some are more useful than others. Obviously, these are unsupported commands and should only be used with great care.

TRACE FLAGS

There are quite a few undocumented trace flags in SQL Server.  However, I’m only going to touch on four today. They are trace flags 3004, 3014 3604 and 3605.

These can be enabled for the current session with the following commands:

– Enable the trace flags

DBCC TRACEON(3004);

DBCC TRACEON(3014);

DBCC TRACEON(3604);

DBCC TRACEON(3605);

GO

These trace flags are going to be used in conjunction with a database restore so we can get an idea about what SQL Server is doing during the process. We’ll be looking at output in the message output screen and output sent to the SQL Server error log.

Before we go much further, I want to briefly explain the purpose of these flags and why it’s important to use them together to get the information we’re looking for.

Trace Flag 3004

Most Database Administrators are aware of instant file initialization. In a nutshell, when instant file initialization is enabled the data files do not need to be zeroed out during creation. This can save an incredible amount of time during the restoration of VLDBs. As you can imagine, the zeroing out of a 1 TB data file can take a very long time.

Trace flag 3004 turns on information regarding instant file initialization. Enabling this trace flag will not make this information available to view. You will still need to turn on trace flag 3605 to send this information to the error log.

Trace Flag 3014

Trace flag 3014 provides detailed information regarding the steps performed during the backup and restore process. Normally, SQL Server only provides a limited amount of information in the error log regarding these processes. By enabling this trace flag you’ll be able to see some very detailed and interesting information.

Trace Flag 3604

Trace flag 3604 can be used under a variety of circumstances. If you’ve ever used DBCC IND or DBCC PAGE then you’ve probably already used trace flag 3604. It simply informs SQL Server to send some DBCC output information to the screen instead of the error log. In many cases, you have to use this trace flag to see any output at all.

Trace Flag 3605

Trace flag 3605 will send some DBCC output to the error log. This trace flag needs to be enabled to see the instant file initialization information made available by trace flag 3004.

Restore without Trace Flags

I created a full backup of the AdventureWorks database and then restored it with the following command:

– Restore the database

RESTORE DATABASE AdventureWorks

FROM DISK = ‘C:\TEMP\ADVENTUREWORKS.BAK’

GO

Without these trace flags the following information is returned to the message output screen:

You can also see very similar information in the error log. I returned the contents of the error log with the following command:

– Read the error log

EXEC xp_readerrorlog;

GO

Restore with Trace Flags

I’m going to perform the restore again. This time my four trace flags have been enabled and I’m hoping to see some additional information in both the message output window and the error log.

In order to make sure the error log is easy to read, I’ve cycled it with the following command:

– Cycle the error log

EXEC sp_cycle_errorlog;

GO

The next step is to execute the restore again.

– Restore the database

RESTORE DATABASE AdventureWorks

FROM DISK = ‘C:\TEMP\ADVENTUREWORKS.BAK’

GO

After the restore is finished, and with the log cleared and the trace flags enabled, we get the following information in the message output window:

It’s easy to see that this output is far more detailed when compared to what we received during our last restore. You notice that there are quite a few additional steps that were not reported when the trace flags were not enabled.

But wait, we also need to check the error log to see what additional information is available there too.

– Read the error log

EXEC xp_readerrorlog;

GO

The most interesting information in the error log is in regard to instant file initialization. You can see that SQL Server is zeroing out the data file during the container prepare process. This means we’re not benefiting from instant file initialization.

Conclusion

It’s important to understand that SQL Server purposely hides this level of detail from us on a daily basis. Arguably, this information isn’t really that necessary. However, if you feel as though you need to have a greater understanding about what’s going on inside the database engine then this is a good starting point on your journey.

Scott Caldwell, SQL Server DBA

I’ve been an Oracle Apps DBA for 6 months now and before that I was a RDBAE junior DBA where I troubleshot MSSQL, Oracle, and Oracle Apps issues.  At RDBAE, I have the great opportunity to seek advice from our Oracle Apps team with decades of experience. Also, they have given me a R12 environment that I can practice on whenever I have time.  Working at RDBAE, I get great opportunities to work on some cool projects.  My first big project was a 24-hour downtime in production to apply the ATG RUP7 patch along with other patching.

With that, I plan to write a discussion and troubleshooting post each week depending on my job load.  Every once in a while I’ll also post a quick blog on helpful commands. I hope you find the posts informative and feel free to post comments at any time.  I’ll post the blogs with the beginning titles:

OraAppYP: Discussion: <title_name>

OraAppYP: Troubleshot: <title_name>

OraAppYP: HelpfulCMD: <title_name>

Enjoy the posts!

Chris Page, Oracle Apps DBA