Finding the Top Resource Consumers Using the 11G DB Console
The intent of this discussion is to identify the easiest ways to identify high resource-consuming SQL using the 11G DBConsole.   In order to prevent this blog from being too large, I’ll attempt to focus my discussion on the topic at hand.   We’ll look at the various tools, advisors and utilities that 11G’s DB Console provides in upcoming blogs.

Once an Oracle database is built, loaded with data and tuned, the only real variables that can affect it on a daily basis is either workload/data volume changes or the SQL statements being executed in it.   Of course, we can always tweak the OS parameters, add hardware, etc., but these changes aren’t made on a daily basis (we hope).  The number of concurrent users can fluctuate depending on peak time periods, economic conditions, etc. and disk utilization can also increase based on a whole host of reasons.

But the real dynamic of any database environment is the access paths that SQL statements take.   Those of us that have been in the database administration field for any lengths of time understand that a single SQL statement’s access path change can transform a database that was providing acceptable performance to one that is “performance challenged”.

Our challenge is then to find the statement that is causing the trouble, determining the resources it is utilizing (or dominating) and identifying the access path that it is taking.    Like its 10G DB Console counterpart, the 11G DB Console makes finding these offending statements much easier than before the consoles were available.   That was when DBAs where DBAs, you had a command line interface.   You had no “SGTs” (sissy GUI tools) available.  The DBA ran SQL statements against the database and pored over reams of output – line by line, by line, by line…..    Trust me when I say that I do NOT classify those times as “the good old days”.   You actually had to take the data “offline” to perform many of the administrative tasks that can be done online today.    Needles to say, I am a huge fan of the SGTs and using them is an absolute requirement here at Remote DBA Experts.

Let’s start our analysis of the 11G DB Console.

We begin our investigation by logging into the 11G’s DB Console.    This home page is not much different than its 11G counterpart.  At the top of the home page we see the tabs we can use to navigate our way through the various console panels.   Like its 10G database counterpart, the 11G database console provides a host CPU graphic in the middle of the screen.     If we click on the host name contained in the CPU graphic or the CPU link in the Active Sessions graphic just to the right of it, the 11G DB Console will respond by displaying the Top Activity Page.

You’ll notice  that the consoles supplied with both 10G and 11G provide numerous links to the Top Activity Page.  We’ll also learn later in this blog that there are links that will allow us to view key database resources and the top SQL statements that are consuming them. There’s a reason for that.   As I stated earlier in this blog, identifying the SQL being executed (and the associated resources the statements consume) plays a key role in any performance problem analysis effort.

The Top Activity Page displays a running history of the top resource consuming SQL statements and sessions.  There is a drop down menu in the upper page that allows us to view the top resource consumers historically.   This historical option finally allows administrators to answer the age-old question posed by various business personnel and application developers “My job ran long two days ago and I think it’s the database’s fault.   What happened?”.     Now we can find out exactly what happened.

When we click on the SQL ID link on the Top Activity Page, the 11G DB Console will respond by displaying the SQL Details Activity Page. The SQL Details Activity Page provides us with information on how active this statement is  (how many sessions are running  this particular SQL statement). The same navigation  links for statistics, activity, plan, tuning history and SQL monitoring are displayed on all of the SQL Details drill down panels.   They allow the user to drill down and analyze information that is pertinent to the SQL tuning process.

You will also notice that the DB Console provides numerous buttons that allow you to activate the SQL Tuning Advisor.  Actually, you’ll find these links everywhere and there’s a reason for that.   Oracle wants you to leverage the advantages that the SQL Tuning Advisor provides.   The more mature these advisors become, the better I like them.    In my next blog, we’ll deviate from our technical discussion and discuss the future of database tuning.  That future will include an increased dependency on automated tools and less of a dependency on a deep knowledge of Oracle optimization and Oracle internals.

One of my favorite panels is the first one on the navigation list.   The SQL Statistics Page provides a wealth of information on the SQL statement we are reviewing.   On the left hand side of the screen, the DB Console is telling us that a SQL Profile is implemented for this statement.  On the right hand side, all of the pertinent resource consumption information is displayed.   Virtually every piece of information needed to evaluate the statements resource consumption is provided including executions, elapsed time, CPU time, buffer gets, disk reads and rows returned.   The console also displays both the total resource utilization for all executions of this statement as well as per execution.   As we learned in previous discussions, if a SQL statement is consuming 300,000 disk reads total but it is run 300,000 times, it probably doesn’t need tuning.

If we click on the Plan navigation tab, the DBConsole responds by displaying information on the SQL Statement’s Access path. I really like the graphical version display that  this tool provides.    In a future blog, I’ll show you how to interpret EVERY SQL statement access path output that I use – from a dump of the plan table to this console display.    If you want to know more information about the particular operation contained in the graphical display, you can click on the operation and the console will respond by displaying a definition of that operation in the Solution Details box.  If you like a more traditional display,  there is a bullet on the left side of the panel that allows you to switch from a graphical depiction of the access path  to a more traditional table display output.

The Plan Control Page displays information on SQL Plan Baselines and SQL Profiles.   We don’t want to delve too deep now into SQL Baselines, but you can rest assure we’ll spend some dedicated time on this feature.   The results that we are receiving with SQL Plan Baselines are making me “cautiously optimistic”.   I love that term.  The optimizer is intelligently deciding to add access paths that are better into the baseline and allowing us to activate them.    All I can say is that it seems to be working as Oracle describes it.

The Tuning History Page provides us with historical information on past tuning changes. When we finally take Oracle’s advice and run the SQL Tuning Advisor, it will display any  profile we implement on this page. If you have implemented several different profiles in the past, DBConsole will provide a listing of them on the Tuning History Page. You then have the option of deactivating one profile and reactivating a past profile if you desire.

Let’s continue our investigation by using the navigation tab on the top left hand side on the Database Home Page to navigate to the Database Performance Home Page. This panel certainly makes it easy for us to determine what resources are being utilized. We can click on the different colors to drill down into specific resources.    Here’s a drilldown page for all of the sessions that are waiting on User I/O.    Once again, the 11G DB Console makes it easy for us to find the individual SQL Statements, as well as the sessions that are the top generators of User I/O waits.  Clicking on the SQL statement on the lower left side of the page will display the SQL Statistics Page. Right back where we started this discussion.   We can easily review the SQL statment’s statistics, access paths, tuning history….

As we have found, the 10G DBConsole provides many different features that allow us to identify the top resource consumers in the database it is monitoring. With time and experience, you’ll be able to quickly determine what sessions; SQL and components are the top resource consumers. In addition, you will also be able to determine what resource the top consumers are utilizing and the SQL they are executing.

In my next blog, we’ll deviate from our technical discussion and talk about the future of database tuning.  Here’s my prediction – The future will not include us being required to become “at one” with the Oracle optimizer.  We’ll be reviewing and implementing decisions made by the intelligent advisors.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

We’ll learn that 10G’s DBConsole and 11G Grid are both able to provide us with all of the information we need to identify the top resource consumers, what resources they are consuming and the SQL they are executing.  In my last blog, we learned how to identify top resource consumers using a couple of very simple SQL statements.  This week, we’ll cover the 10G DBConsole and we’ll end up with a discussion on using 11G Grid Control to identify those poorly performing statements.  That way we’ll have covered the majority of options that are currently available to the DBA community to identify high resource-consuming SQL.

Finding the Top Resource Consumers Using the 10G DBConsole

We begin our investigation by logging into the 10G DBConsole and displaying the Database Home Page. The first indicator that we are having a problem is the Host CPU graphic displayed in the middle of the screen. The DBConsole has placed a small red ‘X’ next to the title ‘Host CPU’ to notify us that CPU is an issue. If we scroll down to the lower half of the screen, we notice that ADDM has already identified the problem and is reporting its recommendations. I’ll discuss drilling down into ADDM recommendations in an upcoming blog. We can certainly use the ADDM output to help us solve our problem but it is important for us to discuss all of the tools that 10Grid DBConsole provides to us.

Let’s continue our investigation by using the navigation tab on the top left hand side on the Database Home Page to navigate to the Database Performance Home Page. This panel certainly makes it easy for us to determine what resources are being utilized. We immediately see that we have a high CPU utilization by the large green area displayed at the top of the screen.

We are able to use DBConsole’s drill down capabilities to drill down to the specific SQL and Sessions that are consuming that resource. It is important to note the DBConsole provides drill down capabilities for all major resources displayed on this page. We activate the drill down for CPU utilization by clicking on the green area displayed on the screen. DBConsole responds by displaying the CPU drilldown panel. DBConsole provides a slider that allows us to go back in time and review the utilization of this resource historically. If we click on the SQL statement drill down on the left side of the screen, DBConsole responds by displaying the SQL Details Activity Page.

Once again, a large green area shows that this statement is consuming a LOT of CPU resources. You will begin to notice that almost every page DBConsole displays during our analysis provides a button to run the SQL Tuning Advisor. I have personally seen the tool executed numerous times and can attest that the recommendations it makes are worthwhile. Please remember this point – the SQL Tuning Advisor often recommends that you implement a Profile that changes the SQL statement’s access path. This tool is currently not a substitute for having a firm understanding of SQL access paths. The advisor will allow you to view the new access path. Review it, determine if you want to activate it and then monitor the new access path for the statement very closely. Don’t blindly implement the profile without monitoring the performance of the new access path.

We can use the navigation tab on this page to navigate to the SQL Details Statistics Page. The SQL Details Statistics Page provides us with a wealth of information on this particular SQL statement. Note the large green area in the Activity By Waits pie chart. The page’s Activity By Time section also denotes both high elapsed and high CPU times. In addition, the page provides us with other pertinent information including the number of times this statement was executed, buffer gets, disk reads, etc… Once again we see a navigation button for the SQL Tuning Advisor.

If we click on the Plan navigation tab, DBConsole responds by displaying information on the SQL statement’s access path. It’s blatantly obvious in this case that we have a Cartesian join occurring. We also see that DBConsole is displaying a Schedule SQL Tuning Advisor button.

The Tuning Information Panel provides us with historical information on past tuning changes. When we finally take Oracle’s advice and run the SQL Tuning Advisor, it will display the activated profile on this page. If you have implemented several different profiles in the past, DBConsole will provide a listing of them on the Tuning Information Panel. You then have the option of deactivating one profile and reactivating a past profile if you desire. Here’s a screenshot of a Tuning Information panel showing multiple executions and profile recommendations generated by the SQL Tuning Advisor. As I stated previously, the intent of this blog is not to instruct you how to use the advisor. I’ve covered that in a previous blog.

Let’s continue our discussion by navigating back to the Database Performance Home Page. If we scroll to the bottom of the page, we can see a column heading titled “Additional Monitoring Links”. If we click on the Top Activity link, DBConsole responds by displaying the Top Activity Page. This is the panel that I personally use the most when I am trying to find top resource consumers in DBConsole. The Top Activity Page displays both Top SQL and Top Sessions. If we click on the SQL ID link provided in each line of the Top SQL section of this panel, DBConsole will display the SQL Details Activity page we reviewed earlier in this blog. We are also able to drill down into more specific session information by using the drill down links provided in each line of the Top Session Panel. As I stated, when I use DBConsole to determine what resources are being consumed, I will navigate to the Database Performance Home Page and look at the graph at the top of the page. Although I may drill down into the resource, I most often find myself dropping down to the Top Activity link to display the Top Activity Page. It allows me to quickly identify both the top SQL and sessions.

There is a link displayed in the middle right hand side of the Top Activity Page titled ‘Run ASH Report’. Since the Automatic Workload Repository, by default, takes snapshots every 60 minutes, performance information could be up to 60 minutes old. As a result, snapshots do not contain enough information to allow administrators to perform analysis on the active workload currently being performed in the database system. Oracle10g contains a new internal utility, called Active Session History (ASH), to provide administrators with access to current performance information. ASH samples data from the V$SESSION dynamic performance table every second and stores the information in V$ACTIVE_SESSON_HISTORY. The information contains the events for which current sessions are waiting. The information pertains to active sessions only; information from inactive sessions is not recorded. The view contains one row per active session for each one-second sample. Administrators are able to access V$ACTIVE_SESSON_HISTORY as they would any other V$ dynamic performance table. DBConsole also provides this handy report to provide you with information derived from statistical data stored in V$ACTIVE_SESSON_HISTORY. The ASH report provides a wealth of top resource consuming components.

The Top Activity Page also allows me to drill down into individual session information. Each Session ID displayed in the Top Sessions chart on the Top Activity page is a link to a set of session information panels. If we click on the Session ID link, DBConsole will display the Session Activity page. The Session Activity Page provides us with information on the resources being used and navigation buttons that allow us to kill the session and enable or disable SQL Tracing. You’ll notice that these buttons are displayed on all of the panels dealing with session information.

If we click on the General Tab, DBConsole will display the General Information Page. The General Information page provides us with a wealth of pertinent information on this specific session. Depending on the application’s architecture, the Client information could help us identify and contact the specific user experiencing the problem. In addition, the panel provides information on the application and server. The most important piece of information displayed on this page is under the heading titled “Wait”. It tells us what resource is currently being waited for, and in this case, the file and object. Great diagnostic information when you are trying to debug a performance problem. The Session Statistics Page is a dump of all the raw statistics for this session. The Open Cursors Page displays information on the SQL statement or statements being executed by this specific session. It provides the text of the SQL statement and drill downs that allow you to access the SQL Details drilldown screens discussed previously in this blog. The Blocking Tree Page helps you determine if this session is being locked out by another process or locking out other processes. Lastly, the Wait Event History Page provides drilldown capabilities into the specific waits this session is experiencing.

Let’s take a look at another set of panels that provides information on top resource consumption. We will navigate back to the Database Performance Home Page and click on the Top Consumers link to display the Top Consumers Home Page. Although I don’t use this page frequently when performing database triage, it does provide some very useful information. My personal favorite is the Top Modules by Service chart on the upper right hand side of the panel. It organizes the resource consumers by the type of service. This information is very beneficial when you have numerous sessions consuming a high level of resources because it allows you to determine if the problem is localized to a specific area of the application.

Currently, it is my SQL*PLUS service that is dominating our database resources. I have seen personally viewed report writer programs, application server programs, TOAD, SQL*PLUS, etc. all displayed on this chart. The information allowed me to quickly identify if the problem sessions were all coming from that one specific component of the application environment. For example, I once viewed TOAD being displayed as the top resource consumer during one of my investigations. I drilled down into the session information, found the accounts that were being used, stopped by the development area and found that a group of developers just had TOAD installed and were testing it by accessing production data. The panel provides drilldowns for Top Services, Top Clients, Top Clients and Top Sessions. The Top Sessions Panel allows me to once again drill down into individual session information. The panel also provides a link that takes you to the SQL being executed by the top sessions.

As we have found, the 10G DBConsole provides many different features that allow us to identify the top resource consumers in the database it is monitoring. With time and experience, you’ll be able to quickly determine what sessions; SQL and components are the top resource consumers. In addition, you will also be able to determine what resource the top consumers are utilizing and the SQL they are executing.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

We’ll learn that both the10G and 11G DB Consoles are both able to quickly provide us with all of the information we need to identify the top resource consumers, what resources they are consuming and the SQL they are executing.   We can also fall back to a couple of SQL statements that I have been using for years.  They still do the job when I need them to.  That task is identifying SQL statements that are high resource consumers.

This week, we will cover my couple of SQL statements, next week, we’ll cover the 10G DBConsole and we’ll end up with a discussion on using the 11G Console to identify high resource consuming statements.

Dynamic Performance Views

The Oracle database maintains a set of views that records current database activity.  The Oracle 11G manual states that these views are “often called dynamic performance views because they are continuously updated while a database is open and in use.”   DBAs often use the terms “V$ Views” or ”V$ Tables” because they are prefixed with the characters “V$”.   Administrators are able to use these tables to retrieve information about activities occurring in the Oracle instance.   The key word is dynamic.   It means that the information is always changing and, based on certain conditions, the information you are looking for may not be available.

There are dozens (and dozens) of V$ views available to DBAs.   They can provide us with a wealth of information on virtually any activity that is occurring in the Oracle instance.    Let’s keep this simple and focus on a couple of tables that can provide us with some pertinent information on SQL statements and the resources they are consuming.   Three of my personal favorite V$ tables of ALL TIME are V$SQLAREA, V$SQLSTATS  and V$SQL_PLAN.  We’ll take a quick peak at V$SQLAREA, V$SQLSTATS in this blog and review V$SQL_PLAN in-depth in upcoming discussions.

V$SQLAREA, V$SQLSTATS, V$SQLTEXT and V$SQLTEXT_WITH_NEWLINES
Let’s take a high level look at some of the views we will use to gather information on SQL statements contained in the library cache. We’ll need this information in later discussions to help us determine what resources the statements are consuming and what access paths they are taking.

• V$SQLAREA - This dynamic performance view contains statistics on SQL statements that are in the shared pool, parsed and ready for execution.   This view is often an aggregate of the contents of the V$SQL view.  We’ll have to remember this when we run queries against the table.   If I’m tuning a statement, I want to see the total impact it has on the environment.   That aggregate information is important to me.   Like any transient memory area, SQL statements in the shared pool (and related statistical information in this table) may get aged out.   If that is the case, we’ll have to wait for the SQL to run again (or run the statement ourselves if we can) to get the statement back into memory and the statistics available in V$SQLAREA.   If the statement is executed a lot, we won’t have any problem finding information on it in V$SQLAREA.   V$SQLAREA displays statistics on shared SQL areas and contains one row per SQL string.   V$SQL’s SQL_TEXT column provides the first 1,000 characters of the text of the SQL statement and a SQL_ID column that we can use to as an identifier when we query the other V$ dynamic performance views. V$SQL’s SQL_FULLTEXT is a CLOB column that contains the entire SQL statement.  V$SQLAREA also provides an abundant amount of information on resource utilization including sorts, memory utilization, disk reads, buffer gets, rows processed.   We will review some of these columns later in this blog.
• V$SQLSTATS - Oracle 10G R2′s data dictionary contains this new view that Oracle describes as being a faster, more efficient way for a DBA to retrieve statistical information about SQL statements contained in memory.   The Oracle manual also states that this dynamic performance view will also provide a greater retention for statistics (even after the statement is aged out) than its V$SQLAREA counterpart.
• V$SQLTEXT – Although V$SQLAREA and V$SQLSTATS does provide a CLOB column that contains the full text of the SQl statement, this view provides an easy way to retrieve the entire text of the SQL statement. If the text of the SQL statement looks like it has been truncated in V$SQLAREA’s SQL_TEXT column, use the value in the SQL_ID column from V$SQLAREA to query V$SQLTEXT to get the entire statement. If the text of the SQL statement contains any newlines or control characters, they will be replaced with whitespace.
• V$SQLTEXT_WITH_NEWLINES – Contains the text of the SQL statement without the newlines or control characters being replaced by whitespace.

Let’s take a look at some of the columns that are contained in V$SQLAREA and V$SQLSTATS.  These tables contain a wealth of resource utilization information!

• SQL_TEXT VARCHAR2(1000) – First thousand characters of the SQL text for the current cursor
• SQL_ID VARCHAR2(13) –  SQL identifier that we can use to identify this statement
• SORTS NUMBER  – Sum of the number of sorts
• FETCHES NUMBER – Number of fetches
• EXECUTIONS NUMBER – Total number of executions
• DISK_READS NUMBER – Sum of the number of disk  reads
• BUFFER_GETS NUMBER – Sum of buffer gets
• ROWS_PROCESSED NUMBER - Total number of rows processed on behalf of this SQL statement
• OPTIMIZER_MODE VARCHAR2(10) (V$SQLAREA ONLY) – Mode under which the SQL statement was executed
• OPTIMIZER_COST NUMBER (V$SQLAREA ONLY) -Cost of this query given by the optimizer
• SQL_PROFILE VARCHAR2(64) (V$SQLAREA ONLY) -SQL profile used for this statement, if any

As stated previously, the information for specific SQL statements contained in the V$ performance views provided above could be flushed from the system based on workload and memory allocations. Although you will always find data for currently executing queries, you may, or may not, find information pertaining to statements that were executed some time in the past. If a SQL statement has been aged out of the library cache, you may not be able to find information on it.

Finding the Top Resource Consuming Queries
If I want to perform a traditional “top down” tuning approach and tune the highest resource consuming SQL, I’ll use the statements below to identify the top resource consuming queries.

The following query identifies the SQL responsible for the most disk reads:

SELECT disk_reads, executions, disk_reads/executions, address, sql_text FROM v$sqlarea WHERE disk_reads > 5000 ORDER BY disk_reads;

The following query identifies the SQL responsible for the most buffer hits:

SELECT buffer_gets, executions, buffer_gets/executions, address, sql_text FROM v$sqlarea WHERE buffer_gets > 10000 ORDER BY buffer_gets;

You can create a more readable report in SQLPLUS by inserting report breaks between the output lines. To generate the report breaks in SQLPLUS, issue the following statement before running the query:

BREAK ON disk_reads SKIP 2 — for the disk read report and
BREAK ON buffer_gets SKIP 2 — for the buffer get report

It’s common knowledge that poorly performing SQL is responsible for the majority of database performance problems. The first query returns SQL statements responsible for generating disk reads greater than 5,000 while the second query returns SQL statements responsible for generating buffer reads greater than 10,000. I used these numbers just as an example but I do personally use them as a starting point from time-to-time. Based on their output, I’ll then adjust the numbers up or down accordingly. The numbers also depend on the system I’m reviewing. I’ll use different numbers for OLTP environments than I would for data warehouses.

You’ll notice that I divide the number of disk and buffer reads by the number of statement executions. If a statement is generating 1,000,000 disk reads but is executed 500,000 times, it probably doesn’t need tuning. But the more it runs, the more emphasis I will place on tuning it. If you can reduce the buffer gets generated by a query that runs thousands of times of day by even small amount, you are still reducing the total workload the system has to process.

Heavy disk reads per statement execution usually means a lack of proper indexing, poor selection criteria, etc.. Heavy buffer reads sometimes means the exact opposite – indexes are being used when they shouldn’t be. But I’m personally most interested in workload, that’s why I always use the buffer hits in my initial queries.  The more times a statement is hitting the buffers, the greater the workload it is exerting upon the database environment.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Introduction
This blog is not about the SQL tuning process. Its intent is also to not teach you to tune, its focus is to help you begin, or brush up, on access paths and SQL tuning. Remember the Foot Rule of Thumb “The mark of being a good DBA is not knowing everything; it’s knowing where to look when you don’t”.

Gaining a well-rounded and deep understanding of Oracle access paths and SQL performance is a wonderfully challenging task. One of the hurdles is the time required to learn how to tune SQL. DBAs are being challenged with ever-increasing workloads, shrinking DBA staffs and increasingly complex technologies to support.   Many DBAs never have the luxury of spending some quality time with the Oracle optimize.   If your shop is like any of the shops I have worked in, your days are spent performing dozens upon dozens of administrative functions that range the spectrum.   From RAC database installs to working with application developers on naming conventions – our workloads seem to increase daily, weekly, monthly….  After spending 12 hours ensuring you have your TODO list done, you don’t have much time to do anything else.

One of my upcoming blogs will be titled “The DBA of the Future”.  A future that is already being shown to us by Oracle.   The feature/functionality of the Oracle database environment will continue to grow.   DBAs will no longer be able to be experts in “silos of one or more Oracle features”.   We will be forced to use the intelligent toolsets that Oracle is creating for us.   Gone will be the days that DBAs will be able to spend dedicated hours gaining a deep knowledge of the Oracle system.   DBAs will need to know a lot about a lot of facets of Oracle technology.   Think about Oracle’s expanding toolsets and technologies: RAC, DataGuard, Fine-Grained Auditing, advanced security options, Streams, compression, partitioning/parallelism, Oracle Fusion….   This list can go on for paragraphs.    How can anyone expect the DBA of the future to be an expert in all of these features?    The answer is – they can’t.    But this is better left for another blog.

In addition, the majority of shops view database administrators as much more than just “table jockeys.” The DBA is often seen as the go-to technician because of their traditionally strong problem solving skills. The DBA is also viewed as the IT staff’s technical generalist because of the working knowledge they have in many different areas of information technology. Those of us that have been working in this profession for any time at all understand that the term “database administration” is really a misnomer. We have to know everything from application and data design to network communications and operating systems (and everything in-between).

As a result, many of us don’t have enough time to dig deep into a single feature of the Oracle database environment. Oracle has recognized this issue and has created the various advisors in 10G to assist us in the monitoring and tuning process. I have written dozens of blogs on the toolsets provided in 10G that are designed to reduce the amount of time we spend administering, troubleshooting and tuning the Oracle Ecosystem (database, operating system, hardware server).   We will cover the 11G tools in upcoming blogs.

The Importance of Understanding Oracle Access Paths
That being said, all DBAs must make an attempt to have a firm understanding of Oracle access paths and SQL tuning. During my career, I’ve learned that there is nothing that can drag down an Oracle ecosystem than poorly performing SQL statements.

That’s when management suddenly doesn’t care about how much you know about RAC, standby databases and data design. They want the database “fixed” and running smoothly again. They suddenly have a single-minded purpose. They quickly begin the chant, the chant that can only be described as the management mantra – “is it fixed yet, is it fixed yet, is it fixed yet, is it fixed yet, is it fixed yet…..”

I’ve actually had my entire chain of command stand behind me in order of where they fit into the management food chain (team lead, project manager, section manager, division manager, VP…) while I was working on a database performance issue. I turned around and chuckled when I saw the lineup of managers. Funny, they didn’t share my humor.

You need to get the performance problem fixed, you’re nervous and you have a 15 page SQL statement on your screen. This is where a strong education in Oracle access paths becomes “somewhat handy.”

SQL Tuning Topics You Must Understand
Here’s a quick laundry list of topics you’ll need to know. All of them are important. I have tried to include resources later in this blog that will provide you with information on all of them. You can use this listing to check off the topics during the education process.

• The Oracle release’s impact on optimization and SQL tuning. Each new release contains features that affect access paths and SQL performance. Sometimes good and sometimes not so good.  You’ll find this information on many Oracle discussion websites.   Most begin with “can you believe this happened?”

• Oracle parameters that affect the optimizer. There is a handful or two of startup parameters that can influence the SQL optimization process. I have provided several articles below on this topic.

• Optimizer modes – Rule, choose, first_rows, all_rows. Each of the modes influence the optimizer to create access paths for the type of workload the database is responsible for supporting. For example, the first_rows optimizer mode may be OK for online transaction processing (read a record/write a record) but probably won’t generate efficient access paths for a data warehouse database where millions of records are summarized.

• Oracle data statistics – How the optimizer uses them, how they affect access paths.

• Oracle system statistics – Current releases of Oracle can also incorporate the system load information during optimization.

• Basic access paths
  • Index only – Oracle is able to read all of the data required to satisfy the query’s data needs from the index structure alone.
  • Index to table – Oracle uses a row identifier to probe the table to satisfy the data request. Why read all of the rows in a table if you can use an index structure to retrieve just the rows you need?
  • Full table scan – Oracle reads all rows from the table. If the statement is going to read the majority of a table’s rows, why would you want it to needlessly traverse an index to get the data? You are reading extra index blocks for no reason. You also need to learn the impact that the high-water mark has on full table scans. Oracle will scan the table to the last block used (as opposed to the last block that actually contains data).

• Join access paths – Used when the statement retrieves data based on matches between two tables (i.e. retrieve all of the employees that have the department name of “Welding”). The employee information is contained in the employee table and the department information (including the department name) is in the department table.
  • Nested loop join – Good path when the join is accessing a small subset of rows.
  • Hash join – Efficient access path for joins that access larger sets of data.
  • Sort merge join – Sorts rows to allow quicker access during the join.
  • Cartesian join – The tables being joined do not have join clauses that relate the two tables together.
  • Outer joins – An outer join returns all of the rows that satisfy the particular join condition and returns additional rows from one table that do not satisfy the join condition.

• Join order – Oracle only joins two tables at a time. If multiple tables are joined, join order also describes the overall order of the tables being accessed. Oracle will join two tables and create an intermediate result set which is then used as input to the next join. Join order plays a significant role in query performance. In general, you want to reduce the number of rows processed as soon as you can during the processing of a given SQL statement. The sooner you can reduce the number of rows being sent to future operations, the faster the query will usually run.

• Subqueries – A select within a select statement. Can be one of the trickier statements to tune, especially when you have multiple subqueries embedded within each other.  I saw 21 subselects embedded in a single SQL statement (along with 1/2 dozen in-line views).   Note to that developer – “just because you can do that doesn’t mean you should.   Doing all the work for an entire application in one SQL statement does NOT showcase your coding skills.”

• Indexes and selectivity
  • B-tree indexes are good for column(s) that have many unique values (high cardinality)
  • Bitmap indexes are used for column(s) that do not have many unique values (low cardinality)
  • How SQL statement predicates can determine if an index can be used. There are times when the way a statement is coded prevents Oracle from choosing an index as the access path. A common problem that often leads to poor performance. You do get a chance to flog the application developer responsible, though.

• Types of index access paths
  • Index unique scans – The SQL statement accesses an index using a column (or columns) that are defined in a unique or primary key index with an equality condition.
  • Index range scans – Oracle scans a set of entries in the index to satisfy a query.
  • Index skip scans – Oracle is able to break down a multi-column index and view them as smaller subindexes. This is achieved by Oracle “skipping” the leading columns in the index and using columns that appear later in the index’s definition.
  • Full scans – Oracle scans all of the index entries. Kind of like a table scan on an index.
  • Fast full scan – Oracle uses multi-block reads to retrieve the index blocks.

• Sorting – Many operations require the database to sort a result set. It could be that the query wants to return the data in a particular order (ORDER BY, GROUP BY). In addition, some joins require that the data be sorted during the operation’s execution. You’ll need to understand why sorts are performed and the impact they have on performance.

• Views – Views can really complicate the tuning process. You think you are accessing a few tables in a query and then find that you are actually joining views together that also contain join operations.

• Hints - One of my previous blogs showed you how to use hints to influence access paths and how hints can be used to educate yourself on the performance of a particular access path operation.

• Bind variables and bind peeking – In the first and second blogs of my next series, I will describe the impact that bind peeking can have on SQL statement optimization. Bind peeking may lead to the predicted access path not matching the access path taken during execution.

• Query transformation – Oracle can rewrite a query during the optimization process. Learn how Oracle uses view merging, predicate pushing, OR expansion and subquery unnesting to attempt to improve execution performance. This will occur on a regular basis and you need to understand how query transformation works and how it affects SQL performance.

• Local predicates vs. join predicates – A local predicate accesses a bind or hardcoded variable (i.e. emp_id = :empid, emp_id = 13344) while a join predicate is used to join two tables together (i.e. emp.dept_id = dept.dept_id)

• Predicate usage – You need to understand how predicate usage affects indexes and access path generation. I have provided information below that discusses predicate conditions (=, >, <, etc) as well as how predicates affect index utilization and access paths.

• Operation selectivity- The number of rows returned by a particular access path operation. Importance of operation selectivity is magnified as the number of tables accessed in the query increases. As stated previously, the sooner you can reduce the rows sent to future operations, the better your query will perform.

• Skewed data and histograms – What happens when you have an index built upon a column in a million row table that has twenty occurrences of the value “OUT OF STOCK” and the rest of the column values contain the value “IN STOCK”? This is an extreme case, but the impact is that even though you may access the table looking for an “OUT OF STOCK” value, Oracle will most likely perform a table scan. You’ll be searching close to a million values, while you need to retrieve only twenty of them. An index would be a much better access path, but Oracle sees that the column has such poor selectivity that it won’t choose it. A histogram identifies skewed data and is able to provide the optimizer with the information it needs to make a more educated decision when choosing between a table scan and index access.

• Parallel processing and partitioning – A best practice for large data stores is to partition data into smaller subsets of data. Partitioning allows data to be broken down into these smaller subsets yet still be viewed as a single-entity by the application. Parallel processing breaks a single request for data into one or more processes that access the data in parallel and return the data to the calling application.

• Parsing – Learn the differences between hard parses vs. soft parses. When the application sends a statement to the database for processing, one of the first steps in the execution process is called a parse. Oracle will check the statement’s syntax, check security, generate the access path, etc… Like most operations the fewer steps it needs to perform the better. You will need to understand the impact that bind variables have on the parsing process and how the hard parse/soft parse ratio affects query and database performance.

Access Path Education
You can start your education on the different access paths that are available to the optimizer by reading Oracle’s Database Performance Tuning Guide that is provided in each Oracle release’s documentation. Before you buy third-party books on any topic, I highly suggest that you read Oracle’s documentation first. The importance of this suggestion bears repeating – READ ORACLE’s DOCUMENTATION FIRST. Here’s a link to the 10G Database Performance Tuning Guide.  Here’s another link to the 11G Database Performance Tuning Guide.   You need to start your education by READING THE DOC!   Trust me on this one.   You have to trust the vendor and it does provide you with a strong foundation upon which to build your education upon.    Third-party information and experience will carry you the rest of the way.

You’ll need to create an account, but its free to register and the process is painless. Virtually every topic that I described above is covered in the Database Performance Tuning Guide. It is very important that you read the guide that pertains to the release that you are working with. Each release contains enhancements to the optimizer as well as new features that affect the optimization process.

Oracle Classroom Education
OK, since I’m an ex-Oracle Instructor, you could have predicted that my next recommendation was to sign-up for an Oracle class on SQL performance tuning. Here’s a previous blog that provides you with a few hints and tips to obtain the most from your classroom experience.

Oracle Education offers a class that focuses on 11G Database and SQL tuning. Here’s the class description provided by the Oracle Education Website.   Oracle also provides a class that specifically focuses on SQL Tuning. If your shop doesn’t have a travel budget for training, there is also a CDROM version you can purchase.

Metalink Notes
Oracle’s premier web support service is available to all customers who have current support service contracts. Oracle MetaLink allows customers to log and track service requests. Metalink also allows users to search Oracle’s support and bug databases. The website contains a patch and patchset download area, product availability, product life-cycle information and technical libraries containing whitepapers and informational documents. A few of the white papers and notes that pertain to SQL tuning, optimization and access paths are provided below (do a search using the note number on the main page in Metalink to retrieve the note):

• 232443.1 – How to identify resource consuming SQL for tuning
• 271196.1 – Automatic SQL Tuning – SQL Profiles
• 199083.1 – Query Tuning Overview – Lots of good links to other articles.
• 398838.1 Frequently Asked Questions. Very good discussion on the optimizer. Discusses queries not using indexes.
• 248971.1 – Query tuning best practices. LOTS of links to other notes. Links discuss a wide range of topics. From parameters that affect optimization to system statistics impact on optimization.
• 35934.1 – Common Issues and Misconceptions about the cost based optimizer.
• 68735.1 Diagnostics for Query tuning.
• 67522.1 – Diagnosing why a query doesn’t use an index. Very helpful.
• 163563.1 – Troubleshooting – Advanced query tuning.
• 372431.1 – Troubleshooting – Tuning a new query
• 207434.1 – Tuning Queries – Quick and Dirty Solutions.
• 100229.1 – Measuring Index Selectivity.  Older note but still pertinent.
• 41954.1 – Hash join operations – Describes one of the more complex join mechanisms.

Third Party Books
If you want to learn how the optimizer works, I highly suggest that you read Jonathan Lewis’s book titled Cost-Based Oracle Fundamentals. It is one of the most educational and informative books I have ever read on the cost-based optimizer.  If I could only recommend one book on the Oracle optimizer, Jonathan’s book would be it. Jonathan also maintains a blog that focuses on a wide range of topics but does include a lot of discussions on Oracle tuning.

Tom Kyte’s Ask Tom Website also provides a lot of information on SQL tuning, access paths and proper coding techniques. One of Tom’s trademarks is to use a snippet of code to reinforce the information he is conveying. I’m a big fan of using examples and a big fan of Tom’s website.

Articles and Presentations
I have learned a lot from this series of articles and whitepapers written By Wolfgang Breitling. Here are a dozen or so performance and availability articles recommended by Oracle. If presentations are more appealing to you, here’s a presentations from various Oracle Open World conferences.

Test, Test, Test
Experience pays. You need to spend time “in the seat” learning how to tune. Read the above information and find a database that you can use as a test environment. You need to work with tables that have small numbers of rows and queries that return small result sets. You also need to run queries that access tables with high numbers of rows and return large result sets.

Don’t’ fall into the trap of favoring one access path or join method for all situations. I once overheard a conversation between a developer that just moved from an online transaction system to the data warehouse team and assist the data warehouse DBA. The developer was looking at an access path and stated “I hate hash joins.” The warehouse DBA stated “Not in this environment you won’t”. All access paths and join methods have a place in Oracle optimization. It’s up to you to learn which ones apply for a given situation.

Thanks for Reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Introduction to Oracle Indexes
Generally, the fastest way to access Oracle data is with an index. The Oracle database contains several different indexing types that are designed to provide complementary performance functionality.

Cardinality vs Selectivity
You’ll hear a lot of technicians use the terms “cardinality” and “selectivity”.   You will also notice that a lot of people use the terms interchangeable (me included).   They aren’t interchangeable.  I’ll have to reread this bog to make sure I don’t!    Cardinality is the number of distinct values for a given column in a table.  A table with good cardinality would be a table that has many unique rows.  The Oracle 11G manual defines cardinality as “The ratio of distinct values to the number of table rows. A column with only two distinct values in a million-row table would have low cardinality.”   The 11G manual defines selectivity as “In a query, the measure of how many rows from a row set pass a predicate test”.   If the selectivity of your predicate is restrictive (a few number of rows are returned based on what you are searching for) it is said to have “good selectivity”.    So you could have skewed data that makes the same query have either high, or low, selectivity – based on the column value that you are searching for.  The query that searches for the value “FOOT” in the last name column (and “FOOT” occurs 10 times in the column being searched on), would have better selectivity than a query that searches for the last name of “SMITH” which occurs a thousand times in that column.

While standard B-tree indexes are most effective for columns containing a high number of different values (high cardinality), bitmapped indexes are most appropriate for columns with a relatively limited number (low cardinality) of different values. The low cardinality statement above comes with several caveats. Instead of spending this entire blog on the bitmap index/low selectivity issue, please turn to Julian Dyke’s presentation titled “Bitmap Index Internals”. One of my favorite blogs on bitmap indexes was written by Mark Rittman. Oracle also provides function-based indexes to allow index access using SQL that contains column manipulations in the WHERE clause. Administrators supporting large data stores use partitioned indexes to decompose large index structures into smaller, more manageable pieces called index partitions. Starting with the 8i release, Oracle places index data in the separate index partitions based on the index’s partitioning key.

Before we begin our reviewing some of the more popular Oracle index types, let me provide you with a few thoughts on indexes in general.

How Many Indexes Can I Build?
This subject has always been a matter for great debate. The DBA must balance the performance of SELECT statements with their DML (INSERT, UPDATE and DELETE) counterparts. SELECT statements that return a limited number of rows from a large table yet access non-indexed columns will suffer from poor performance. Conversely, if you have too many indexes on a particular table, DML statements may be adversely affected.

The DBA must take the business requirements, application processing workload and workload scheduling into consideration when determining how many indexes to build. If you compare the performance improvements an index makes on a SELECT statement to the negative affect it has on DML statements, you will find that the benefits of building the index usually far outweigh the performance drawbacks.

Indexes on columns in the WHERE clause of SELECT statements can reduce query times by minutes and even hours. The creation of additional indexes may add additional time to on-line transactions that execute DML statements. Additional indexes will have the greatest negative impact on DML statements that access a large number of rows. The more rows that are inserted, deleted or changed, the greater the negative impact will be. Traditionally, programs that process large volumes of rows are scheduled to execute during off-hours.

The DBA must also consider the needs of the business. What process is more important to the business unit – getting the data in or getting the data out? Who complains the most? Is it the business user that must wait minutes for their transaction or hours for their report to retrieve data or the business user that is waiting an extra few seconds for their update transaction to complete? Is the nightly batch window tight on time?

The DBA will need to find out how much time the additional indexes add to programs that process large volumes of rows. In addition, the DBA must determine when these programs run. If they run at night or do not require high-performance, consider building the index.

If the transaction update performance requirements are excessive (dot com applications are one example), keep the number of indexes to a minimum.

Index Monitoring
Determining if an index will increase performance is a pretty straightforward process. The administrator is focusing their tuning efforts on a particular query and is able to gather the specific information necessary to assist in the decision making process.

Dropping unused indexes is also an important part of application tuning. We learned previously that indexes force Oracle to occur additional I/O every time a row is inserted or deleted into the table they are built upon. Every update of the table’s columns incurs additional I/O to all indexes defined on those columns. Unused indexes also waste space and add unnecessary administrative complexity. Since unused indexes (excluding those used to enforce integrity constraints) do not add any benefits to the environment, why keep them?

Determining if indexes were being used in releases prior to Oracle9i was a time consuming and error-prone process. EXPLAIN plan and trace output could be used but there was no single mechanism that monitored index usage at the database level.

Starting with release 9i, the Oracle database simplifies the index usage monitoring process by providing the ALTER INDEX……… MONITOR USAGE command. The statement below turns monitoring on for the index SCOTT.EMPIDX while the second statement ends the monitoring session:

ALTER INDEX scott.empidx MONITORING USAGE;
ALTER INDEX scott.empidx NOMONITORING USAGE;

The V$OBJECT_USAGE table can then be accessed to determine if the index was used during the monitoring session. When the session is started, Oracle clears the information in V$OBJECT_USAGE for the index being monitored and enters a new start time identifying when the index monitoring session started. After the index monitoring session is concluded, the USED column in the V$OBJECT_USAGE table will contain the value ‘YES’ if the index was used during the monitoring session and the value ‘NO’ if it was not.

Parameters that Impact Index Usage
The parameters listed below influence the Oracle cost-based optimizer to favor or not favor index access. Please note that this is not an all inclusive list. It is a listing of parameters that I feel have the greatest chance to influence the optimizer to choose, or not choose, index access paths. We’ll use these parameters during our scientific analysis.

• OPTIMIZER_MODE = first_rows or first_rows_nnnn – The optimizer chooses the best plan for fast delivery of the first few rows or the first nnnn rows. The first_rows_nnn replaces the first_rows parameter in later Oracle releases. The first_rows is available for backward compatibility. More often than not, that access path will include an index. This optimizer mode tends to favor nested loop joins over hash and merge scan.  It is important to note that using this mode is not a switch. It won’t change each and every table scan and hash join to index access and the nested loop join method. The optimizer, at times, will favor index access and nested loop joins. The reverse goes for the all_rows optimization mode below.
• OPTIMIZER_MODE = all_rows – The optimizer chooses the best plan for fast delivery of all of the rows that queries return. The optimizer may decide to choose a full table scan over index access and hash joins instead of nested loop.
• OPTIMIZER_INDEX_COST_ADJ = xxxx – This parameter lets you tune the optimizer to be more or less index “friendly.” It allows the administrators to influence the optimizer to make it more or less prone to selecting an index access path over a full table scan. Here’s an excellent series of blogs by Richard Foote on this parameter.    I have used this parameter a few times when I was freezing access paths using stored outlines.  More on that later.
• OPTIMIZER_INDEX_CACHING = xxxx – You set this parameter to a value between 0 and 100 to indicate the percentage of the index blocks the optimizer should assume are in the cache. Setting this parameter to a higher value makes the indexes on the inner table of a nested loop joins look less expensive to the optimizer.  Richard Foote’s blog on this parameter is excellent.

Index Affects on Access Paths
If a table only contains a few hundred rows, queries may run faster if the optimizer chooses to read all of the blocks in the table as opposed to using an index. The I/O generated traversing the index blocks to get to the table row entries would be higher than if Oracle read just the blocks allocated to the table being accessed.

What if we access a larger table using a column with few unique values (poor cardinality)? As we learned previously, cardinality describes the number of different values stored in a column.  If our statement contains a WHERE clause that searches for a column value that is contained in 90% of the table’s rows, it is best that Oracle, once again, read each and every row in that table. Conversely if a WHERE clause searches for a column value that appears in 1% of the table rows, it would be beneficial for the optimizer to choose an index.

Here are some examples of index access paths that we will see during our testing:

• Index only – Oracle is able to read all of the data required to satisfy the query’s data needs from the index structure alone.  An old DBA trick (from the mainframe days), is to add the columns that are used in the SELECT clause behind the columns in the index that are used in the WHERE clause.   The query you are attempting to tune will then stay within the index structure and not be required to access the table.
• Index to table – Oracle searches through the index structure looking for the key value(s), then uses row identifiers to probe the table to satisfy the data request.
• Index unique scans – The SQL statement accesses an index using a column (or columns) that are defined in a unique or primary key index with an equality condition.
• Index range scans – Oracle scans a set of entries in the index to satisfy a query.
• Index skip scans – Oracle is able to break down a multi-column index and view them as smaller subindexes. This is achieved by Oracle “skipping” the leading columns in the index and using columns that appear later in the index’s definition.
• Full scans – Oracle scans all of the index entries. Kind of like a tablescan on an index. Oracle drops down to the leaf blocks and traverses the leaf blocks using the leaf pointers.
• Fast full scan – Oracle uses multi-block reads to read both leaf and non-leaf blocks. Non-leaf (branch blocks) are discarded.

But indexes influence more than just index-related access paths. Indexes can also impact the type of join operations used. Here’s an “over the top” example to clarify. If you are joining two tables together in a SQL statement on join columns that have good selectivity (returns relatively few rows compared to the table size), Oracle will favor index access paths and a nested loop join. If you don’t have indexes on the join columns, Oracle may choose table scans using a hash join instead of the nested loop join method.

Index Types
Let’s continue our discussion by reviewing some of the more popular types of indexes: B-Tree, Bitmap and Function.

B-Tree Indexes
A traditional B-Tree index stores the key values and pointers in an inverted tree structure. The key to good B-Tree index performance is to build the index on columns having a lot of different values.  Oracle is able to quickly bypass rows that do not meet the search criteria when searching through indexes built on columns having a high degree of cardinality.

Bitmap Indexes
As I stated previously in this blog, there is a sense of confusion about bitmap indexes and the benefits they provide to columns with low cardinality. Instead of spending this entire blog on the bitmap index/low cardinality issue, please turn to Julian Dyke’s presentation titled “Bitmap Index Internals”.

The optimizer can be stubborn at times. It can be particularly stubborn when you want it to choose a single bitmapped index for an access path. A single bitmap index may not be chosen at all. The optimizer will be more inclined to choose bitmapped indexes as an access path if it can use multiple bitmapped indexes simultaneously. That’s where the benefits of bitmaps are realized.

Bitmap Indexes and Concurrency
Anyone accustomed to database programming understands the potential for concurrency problems. When one application program tries to update data that is in the process of being changed by another, the DBMS must sometimes forbid access until the modification is complete in order to ensure data integrity.

Each entry in a B-Tree index entry contains a single ROWID. When the index entry is locked during an update, a single row is affected. A bitmap lock affects a range of entries which could have a negative impact on other transactions attempting to update rows already locked.  Julian explains in his presentation that updates to Bitmap indexes in 10G may not be as expensive at they were in earlier database releases.

Locking issues affect data manipulation operations in Oracle. As a result, bitmapped indexes are not always appropriate for OLTP applications that have a high level of concurrent insert, update and delete operations. Concurrency is usually not an issue in a data warehousing environment where the data is maintained by bulk loads, inserts and updates.

Function Based Indexes
Oracle 8i solved an indexing problem that had been affecting database performance for close to a decade. Before Oracle8i, any SQL statement that contained a function or expression on the columns being searched on in the WHERE clause could not use an index.

For example, the statement:

SELECT * FROM employee_table
WHERE Upper(first_name) = ‘CHRIS’;

would not use an index. A full table scan would be required to retrieve the desired result set. We now know that we are able to use B-tree and bitmap indexes to speed query performance.

In Oracle8i and later releases, we are able to build both bitmap and B-tree indexes on columns containing the aforementioned functions or expressions. The following index could be used to increase performance of the query:

CREATE INDEX upper_first_name on employee_table (upper(first_name));

Function based indexes will affect the performance of DML statements that manipulate the columns contained in the function based index. The more complex of the expression used, the more time the database will require to update the index.

11G Invisible Indexes
11G introduced invisible indexes. They aren’t totally invisible as we can learn from Richard Foote’s blog on invisible indexes. Richard’s blog entry points you to another blog entry that contains a demo on invisible indexes. Please take a look at this information. You can see by the date that it is a bit old but the information is useful.  As I stated in an earlier blog, I will point you to websites that I feel can be trusted.  Richard Foote is the Oracle index expert and I visit his blog often.

I think that invisible indexes do provide benefits to DBAs who want to build the index but have the opportunity to “turn it off” and “turn it on”  when desired.

There are folks that spend dozens of hours learning about access paths and the affects that indexes, hints, statistics and parameters have on Oracle optimization.  The positions that I have held (except for my stint as an Oracle instructor), always pulled me in several non-technical directions that prevented me from spending as much time learning the internals as I would have liked.  I always seemed to be getting phone calls from managers asking me to make sure my staff had their new database built by the end of the day, getting everyone ready for the next big application turnover, running server consolidation/server deconsolidation, server migration, new application implementation projects.  That and all of the HR activities that surrounds DBA managers and DBA management.  There’s nothing like those phone calls from your local application department Vice President telling you “one of your DBAs is scaring my application developers”.

Wrapup
The intent of this blog was not to provide you with an all-inclusive education on Oracle indexes.  We are laying the groundwork to begin our own scientific analysis of Oracle access paths.   We want to begin testing our own hypothesis.    We can rely upon Richard and Jonathan and Mark to provide us with a base of information but the intent of this lesson is to learn for ourselves.  As I stated previously, we need to begin experimenting on our own to fully understand the Oracle optimization process. In my next blog, we’ll use the information we learned in this series to start our experimentation.
Thanks for reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

In my last post, we reviewed the parameters that we will be modifying to influence the access paths. In this blog, we’ll take a look at the hints we can use to ask the optimizer to select an access path that is different than the one it would normally choose.

Additional Sources For Tuning Information
There will be times when I point you to different sources of information. The sites I will send you to are the sites I trust. First and foremost is the Oracle documentation. You need to READ THE MANUALS!!! Sorry, just reverting back to my life as an Oracle instructor. During class I would often pound the desk and yell out – “YOU MUST READ THE MANUALS!!!!”. They really do contain a wealth of information on the database environment you are administering including SQL performance and the tuning process. I understand that many of you want the “abridged” and shortened version to get just the information you need, but the manuals have come a long way and really are quite beneficial. They are much more to the point and are very good at giving you the pertinent information you are looking for.

So here is my first reading assignment. It’s a section of the 11G Performance and Tuning Guide that discusses optimization. If you read this section, you will have a VERY FIRM FOUNDATION of how the optimizer works and factors that influence the optimization process. The information provided covers everything from statistics to access path descriptions. You need to spend the 40 minutes or so reading this if you want to have a better understanding of the Oracle optimizer. The only really complex discussion is on Transforming Queries. The rest of the information is easily understood and just as clear as any third-party offering. We’ll cover query transformation in a couple of weeks.

I will also send you to various other websites. These folks (Tom Kyte, Jonathan Lewis, Hotsos crew, etc.) have spent countless hours deep in the internals of the database. There is really no need for me to take information that is accrurately presented in an easy-to-read form and regurgitate it.

So, let’s take a look at SQL Hints and how we can use them to investigate different types of access paths.

SQL Hints
Administrators embed hints in a SQL statement to influence the optimizer to choose a particular access path.

By using hints, you are telling Oracle that your access path is better than the one the optimizer is choosing. It’s a safe assumption that most of us aren’t as smart as the optimizer. Let it make the choice, unless you are certain the optimizer is choosing the incorrect access path.

But what happens if the optimizer is making incorrect decisions? Before you begin adding hints to SQL or freezing access paths using Optimizer Plan Stability or 10G Profiles, consider taking the following steps first:

• Determine if it is actually an incorrect access path that is causing the performance problem. It may be some external influence affecting the SQL (hardware, workload, and so on).
• Identify and review the SQL taking the bad access path for proper SQL coding techniques.
• Verify that statistics have been generated on the tables and indexed columns.
• Review parameters that affect SQL optimization (optimizer_mode, optimizer_index_cost_adj, optimizer_index_caching, optimizer_dynamic_sampling, optimizer_features_enable, optimizer_max_permutations).
• Investigate system statistics. Is it activated? Is it configured correctly if it is activated? Should it be activated?
• Does the application use bind variables? If so, investigate bind peeking quirks.
• Check for skewed data. Review histograms and their effect on skewed data.
• Go to Metalink and review optimization bugs for your release. Oracle could have already identified your issue and fixed it. OK, so you have performed all of the actions cited previously and you find that the optimizer is actually making an incorrect decision. Regardless of what some industry pundits may tell you, the optimizer is NOT infallible; it can make mistakes. Oracle created hints for a reason, and wouldn’t have made them public if it didn’t think we really needed them from time to time. If you are forced to add hints to the query to improve its performance, do so intelligently and judiciously.

Using Hints to Compare Oracle Access Paths
OK, now that I have provided you with my standard warning on hints, the intent of this blog is to learn how to use hints to influence access paths for testing purposes. Using hints will allow us to evaluate the effect that different access paths have on SQL statement performance. We will run the statement without any modification, review the access path and performance statistics, use a hint to (hopefully) change the access path, run the statement again and compare the before and after results. Since hints can be embedded in virtually any SQL statement, they will provide us with an easy mechanism to learn more about access paths. We are on our way to becoming database performance scientists!

We’ll begin our access path scientific analysis by using a very basic set of hints to influence the optimizer to choose a different access path. The hints I will be using in my introductory demo are:

• Hints for optimization mode – We will be asking Oracle to optimize the statement using different optimization goals. Since we are using Oracle9i, Oracle10G and 11G for my demos, we’ll be asking it to choose, first_rows, all_rows and rule. We’ll be using Oracle9i to show the rule hint only (since I already have one set up) and then get back to 10G/11G.
• Hints for access paths – Access path hints ask the optimizer to choose the access path it recommends. We’ll be asking Oracle to use an index that it didn’t choose in the original access path it generated. We’ll also be asking the optimizer to choose a full table scan instead of using an index.
• Hints for join operations – Oracle provides several different join methods for statements that join one, or more, tables together. We’ll ask the optimizer to choose nested loop, merge scan and hash joins.
• Hints for join order – Oracle only joins two tables at a time. If multiple tables are joined, join order also describes the overall order of the tables being accessed. Oracle will join two tables and create an intermediate result set which is then used as input to the next join. Join order plays a significant role in query performance. Both in the outer and inner tables selected and the overall join order. In general, you want to reduce the number of rows accessed as soon as you can during the processing of a given SQL statement. The sooner you can reduce the number of rows being sent to future operations, the faster the query will usually run. We’ll ask the optimizer to choose different join orders to determine the impact it has on SQL performance.

The bullets above are just a subset of all of the hints that are available. For a complete listing (and their definitions), please refer to the Oracle Database Performance and Tuning Guide for your release. You’ll find the documentation on Oracle’s Technet Website.  Here’s a link to the 11G documentation on hints.  Before continuing this series, I highly suggest you read this section of the 11G documentation that I provided earlier in this blog that describes the different access paths.

Recommended Toolsets
You need to be careful about the tools that you use when you are using access path information to tune queries. Some tools, like the old-fashioned “explain plan into PLAN_TABLE” and SQL*PLUS Autotrace are predictions of the access path a given query will take while V$SQL_PLAN and SQL TRACE output is the access path actually taken during statement execution. A lot of this has to do with bind variables and bind peeking. We will definitely learn a lot more about access paths changing at runtime. But for the sake of this discussion, let’s operate under the premise that one of the ways we can reduce the likelihood of an access path change occurring is to replace the bind variables with hardcoded values when we run the explain. Which is exactly why we will be using hardcoded values for most of our tests.

Demo Document
The intent of this demo is not to train you to identify which access path is most optimal for a given situation. Its intent is to help you gain experience interacting with the Oracle optimizer. Take it from your friendly ex-Oracle instructor, spending time experimenting with the optimizer and analyzing the performance statistics that different access paths generate is critical to your tuning education. There really is no substitute for time spent “in the seat” performing your own scientific analysis on query optimization.

The demo will show you how to use hints and the ALTER SESSION SQL statement to influence the optimizer to take a different access path than it normally would. You can then compare the access paths and their associated performance statistics to obtain a better understanding of what accces path is best for your test queries.

Here’s a link to the Hint Demo Doc.   It is in Word format for ease of reading.

Now that you understand how hints can be used to influence access paths, you can use this information to assist you in determining how the different access paths change query performance. You can retrieve a SELECT statement that is running in the database, use hints to influence the access path and see the changes the different access paths have. We’ll learn more about selecting representative SQL statements in upcoming blogs.

In my next blog, we’ll discuss what SQL statements to use for your testing and what to look for when you compare the results.

Thanks for reading,

Chris Foot
Oracle Ace
Director Of Service Delivery

Introduction
The objective we are trying to accomplish is to identify the effects that startup parameters, statistics and hints have on access paths, and ultimately, SQL performance. There really is no substitute for spending time “in the seat” learning how different environmental settings influence the optimization process. You also need to spend some time changing a statement’s access path and noting the impact that different access paths have on performance.

I have found that most good tuners share a common set of traits. They are inquisitive by nature, understand that there is no substitute for experience and dedicate lots of time performing scientific analysis on SQL statements and database performance.

A well known tuning guru, Jonathan Lewis, uses terms like “scientific analysis” in his discussions on SQL and database tuning. It’s really a very appropriate description of his activities. After reading many of his works, I would describe him as a database performance scientist. He identifies something he wants to learn more about, creates a test environment, executes a series of tests, analyzes the data and provides documented results to reinforce his theories.

That’s what we all MUST DO to expand our knowledge on the Oracle optimization process. We need to become database performance scientists. We do that by creating a test environment, running and documenting performance baselines, changing the environment to influence the statement’s access path and documenting the results.

We’ll begin this series by learning how to select a test environment and document it. In upcoming blogs, we’ll discuss the tools we will use to measure and compare our results, select a set of SQL statements to use for our testing, create our test cases, execute the tests and document the results.

Setting up a Test Environment
Running test cases to identify the effects that different database environmental settings have on a statement’s access path is not as monumental as it may seem.

Most production databases have a test counterpart. Select an environment that is actively used by your application developers. Don’t worry, if one of our tests causes a statement to “run longer than anticipated”, we can use the trusty ALTER command to kill our session. We can also run our workloads during lull times. Lastly, the majority of changes we will make to influence the statement’s access path will be done in our own session. We won’t be making changes that affect the entire database environment.

I prefer to use an active test environment because it allows me to easily select SQL statements to use as test cases, the data is usually more representative of what will be found in production, and the developers will most likely have a firm understanding of the active workload being executed.

You need to talk to the developers who are responsible for running workloads on that test environment to ensure that the changes being made to the test data don’t skew your results from one test execution to the next. You can’t run a test, have a load insert another 100 thousand rows in the table, run another test and expect to have a good comparison. You want your test bed to be active but not so active that it complicates your testing process or causes your tests to generate incorrect results. You’ll need to use common sense when selecting the test environment as well as determining the most optimal time to run your test cases.

We also want to choose a test environment that is providing adequate performance for the workloads being executed upon it. We really don’t want to use an environment that isn’t performing well to begin with.

There’s a myriad of options available to you. If you don’t want to impact any of your test environments, create a test environment of your own. Clone one of your test databases to a sand-box environment if you can.

You’ll also want to make sure that statistics are up to date on the database you will be using as a test bed. If you are running 10G/11G databases, the database will run statistics jobs for you automatically (isn’t 10G/11G great?). If you don’t have statistics run automatically, it will be up to you to analyze the data objects to ensure that the statistics optimally represent what is stored in the data structures.

Documenting the Parameters That Affect the Optimization Process
The next step is to document the environment. There are a couple of dozen parameters that affect optimization and SQL statement performance. To begin, we are going to choose the basic parameters that are easy to change and have the biggest impact on optimization. These are not all of the parameters that can influence the optimization process, just the ones that are easy to change and provide the best chance of successfully achieving an access path change.

It is important that we read the documentation beforehand for these parameters for the specific Oracle release that we are using as our test environment. We know that each Oracle release may contain enhancements to these parameters that change the effect they have on the optimization process and how we alter them to different values.

We’ll want to document the following parameters to begin our scientific analysis:

cursor_sharing – For our first set of initial tests, we’ll hardcode values in our selection criteria to ensure that our statements aren’t affected by cursor sharing.   We will review cursor sharing and the impact it has on statements using bind variables, in-depth in upcoming blogs.

db_file_multiblock_read_count - The number of blocks that Oracle will read in one I/O when performing a scan of data (i.e. table scan).   We need to be aware that in 10G/11G that Oracle will choose the optimal size of this parameter for us and could adjust it based on database workload.

optimizer_features_enable - This parameter allows you to make the optimizer behave as it would for a specific release of the Oracle database. You set the value to a release identifier (the listing of the optimizer releases that you can set this parameter to is provided in the Reference Manual) and the optimizer will act as it would if the database were at that release.

optimizer_index_caching – Administrators will often set this parameter in conjunction with optimizer_index_cost_adj to influence the optimizer to use more indexes and Nested Loop joins. Setting this parameter makes Nested Loop joins look less expensive when compared to Hash or Sort-Merge joins.

optimizer_index_cost_adj - Allows the administrator to make index access more, or less, costly to the optimizer. The default value of 100 tells the optimizer to evaluate the index using the regular cost. The lower you set this value, the less costly the indexes will look to the optimizer. Conversely, the higher you set this parameter, the more expensive the indexes will look.

optimizer_mode - Sets the approach the optimizer will use when analyzing queries. Since there have been a few changes made between Oracle 9i and Oracle10g, I’ll provide information on both sets.

optimizer_use_pending_statistics -   New  parameter (and feature) to  Oracle Database 11g.  In previous releases,   when you gathered optimizer statistics, the statistics were automatically published once the gather was completed. 11G provides administrators with the option of separating statistics gathering from statistics publishing.  This feature allows us  to test the newly gathered statistics before they are published.

optimizer_capture_sql_plan_baselines and optimizer_use_sql_plan_baselines – Another new 11G feature that we will be testing in the near future.   When these two parameters are set, the 11G  optimizer will look for a SQL plan (access path) baseline for the SQL statement being optimized. If one is found in SQL Management Base, then the optimizer will review the data access costs of the plans and pick the one with the lowest cost.

Oracle 9i provides the following settings:

Rule - The optimizer will use the rule-based approach during analysis. The rule-based optimizer does not use data statistics as input when generating the access path. Each access path is assigned a numerical ranking. The rule-based optimizer chooses the access path that has the most favorable numerical ranking. The rule-based optimizer has been superceded by the cost-based approach. There are a few cases where I have seen the rule-based optimizer choose a better access path than the cost-based method – but not many.

Choose - If optimizer_mode is set to choose, the optimizer is able to switch between rule and cost-based optimizations. When optimizer_mode is set to CHOOSE, the optimizer uses the all_rows cost-based approach for a SQL statement if there are statistics in the dictionary for at least one table accessed in the statement. If you generate statistics on one table, every query that accesses that table will use the cost-based optimizer. What happens if other tables in the query do not have statistics collected? The optimizer will make an educated guess on the statistics for those tables. The problem is that Oracle isn’t always a good statistics guesser and the end-result is a “less than optimal” access path.

first_rows - Influences the optimizer to choose an access path that minimizes response time. Most often used for online transaction processing systems that return small result sets. The optimizer favors Nested Loop joins and index access.

all_rows - Influences the optimizer to choose an access path that minimizes total execution time. Most often used for decision support and data warehouse environments. The optimizer tends to favor full table scans, Hash and Merge-Scan joins.

Oracle10g/11g settings for optimizer_mode:

first_rows - The same influence on the optimizer as it did in Oracle9i.

first_rows_n - Where “n” = 1, 10, 100, 1000. Influences the optimizer to optimize queries to provide the fastest response when returning the “n” number of rows. Acts as a throttle, which allows you to better balance the optimization process.

all_rows – The same influence on the optimizer as it did in Oracle9i.

We will use the ALTER SESSION SQL statement to alter these parameters during our scientific analysis on the effects they have on the Oracle optimization process.

Documenting Our Test Tables
After we document some of the parameters that affect optimization, let’s turn our attention to documenting the tables we will be accessing. In my next blog, I’ll provide you with a few hints and tips on how to select or create SQL statements to use in your test cases but let’s continue our discussion on documentation.

The following information will provide you with a good base of information on the data objects our statments will be accessing. Since we are just beginning our scientific analysis, we’ll use basic storage objects (tables and b-tree indexes). We’ll discuss some of the more complex objects (bitmap indexes, partitioning, etc.) in later blogs.

Row counts for all tables that are accessed by our test queries can be found in the num_rows column in dba_tables if we have statistics generated for our tables.

Number of blocks the table is using can be found in the blocks column in dba_tables if we have statistics generated for our tables.

Index listing for all indexes on our tables. The query below will provide you with a listing of indexes for a given table:

select b.index_name, b.column_name, a.uniqueness, b.column_position
from sys.dba_indexes a, sys.dba_ind_columns b
where a.index_name = b.index_name
and a.table_owner=’&table_owner’ and
a.table_name = ‘&table_name’
order by b.index_name, b.column_position
/

Once we find all of the indexes and columns in those indexes, let’s check them for both selectivity and data skew. Selectivity is the number of unique values for a given column. Skew means that some values can occur a few times in a column while other values can occur many, many times. Since data skew will affect optimization, that information will also be important to us.

We can easily find the selectivity for a single or multi-column index by accessing the distinct_keys column in our dba_indexes table if we have statistics generated. For multi-column indexes, we will want to check the individual selectivity for each column in our multi-column index. We can do this with the following query:

select count (distinct index_colname) from owner.table_name;

Where index_colname is one of the columns in our multi-column index,owner is the table owner and table_name is the name of our table. We’ll need to do this for all columns in our multi-column indexes.

We’ll use the following queries to identify data skew and find out some information on histograms:

select index_colname, count(*) from owner.table_name
group by index_colname;

Where index_colname is the column name, owner is the table owner and table_name is the name of our table. We’ll need to do this for all columns in our indexes. We’ll do this for both single column and multi-column indexes.

select * from dba_tab_histograms where owner=’&owner’ and table_name=’&table_name’;

Where owner is the table owner and table_name is the name of our tables. We’ll need to do this for all tables that our queries will be accessing.

Coming Up Next
We’ll continue our discussion in the next blog when we review the hints we will be using to influence the optimizer’s choice of access paths. In addition, we’ll discuss the various tools we will use to measure and compare our results.

Thanks for Reading,
Chris Foot
Oracle Ace
Director Of Service Delivery