1. Oracle Services on RAC: Five
Things You Might Not Know
ARDENTSALES.LAB.ARDENTPERF.COM =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rh4lab10)(PORT = 1521))
(CONNECT_DATA = (SID = ORCL))
)
ARDENTACCT.LAB.ARDENTPERF.COM =
(DESCRIPTION =
(ADDRESS_LIST=
(FAILOVER = on)
(LOAD_BALANCE = on)
(ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab12-vip)(PORT = 1521))
(ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab13-vip)(PORT = 1521))
(ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab14-vip)(PORT = 1521))
)
(CONNECT_DATA= (SERVICE_NAME = orcl.lab.ardentperf.com))
By Jeremy Schneider )
T
he use of Services is fundamental to increasing The manuals give us the first clues that there is more to services than meets
the eye. One example is the “Service Registration” illustration from chapter 5
availability and performance of Oracle Databases. This of the Oracle Database Net Services Administrator’s Guide. It illustrates
is true for both RAC and non-RAC databases, although independence between listeners and databases. Databases can register
with multiple listeners and listeners redirect connections to multiple
services are especially important in RAC. In this article, we databases. The service is the glue holding this together: The PMON process
learn five things about services that are not well understood for each database instance registers its service names with one or multiple
listeners. This is illustrated in the figure below.
and see how they can help.
Every application wants to be a high-performance, zero-downtime application
when it grows up. And like our teenagers, applications are growing up faster
these days. Whether driven by industry buzzwords or real requirements, it
seems like every DBA I know wants to dramatically increase availability and
performance on their systems. In Oracle Databases, the use of services is
fundamental to meeting these challenges. This is true for both RAC and non-
RAC databases, although services are especially important in RAC.
Since their introduction in the late ’90s services have steadily become more
and more advanced. Oracle Database 10g saw the greatest growth in services-
related functionality and today they touch almost every aspect of database
management. Understanding services is a basic, core responsibility of anyone
who plans or maintains Oracle cluster database systems. Nonetheless, it is
surprising how often services are overlooked or misunderstood! As we will
see in this article, there is a lot more to services than meets the eye.
Before we begin our deep dive, let us state that since services apply to both
RAC and non-RAC environments, we will discuss both aspects in this article, Illustration from Oracle Database Net Services Administrator’s
although we will focus on RAC. When applicable to single-instance non-RAC Guide, Chapter 5
environments, we will point that out specifically. However, there is a lot more to services than just connecting clients to
databases. They are also an important aid for performance tuning, resource
1. Services Are a Lot More Than Just a Connect Method management, and job control.
What is the first thing you think of when you hear the word “services” with One of the most important performance tuning tasks is capturing the right
Oracle? For most database administrators – even RAC DBAs – the first thing historical and current performance data – and services are an integral part of
that comes to mind is the TNSNAMES file. Or perhaps LISTENER.ORA and the Oracle Database 10g statistics collection. The core dynamic views V$SESSION
output of the lsnrctl utility come to mind. Version 8 of Oracle was the first and V$SQL report service information. Wait event information is recorded in
version to introduce the SERVICE_NAME clause for the CONNECT_DATA V$SERVICE_EVENT and then rolled up into V$SERVICE_WAIT_CLASS. These
section of TNSNAMES – and Oracle immediately recommended using it rather service-level statistics are automatically collected by the database and stored
than SID. However, at the time it was not immediately clear what the in the AWR. Furthermore, alerts can be defined for specific warning and
advantages were—particularly in a single-instance environment. critical response time thresholds by service. The instrumentation even
extends to database control and grid control where there is a “Top
continued on page 14
3rd Qtr 2008 n Page 13

2. Oracle Services on RAC: Five Things You Might Not Know continued from page 13
Consumers” dashboard showing various metrics of each service at a glance If you are not using RAC (or MTS/Shared Servers), then getting started with
relative to the others. A sample screenshot is shown below. services is as easy as setting a single-instance parameter―and it does not even
require a database restart! All you have to do is assign a comma-separated list
of service names to the initialization parameter SERVICE_NAMES and Oracle
will automatically create each service on the fly. Each service name that does
not have a domain gets the DB_DOMAIN, and the PMON process will
immediately register each of these services with the listener. (It will blindly
try port 1521 by default; if your listener is on a different port, then set the
LOCAL_LISTENER parameter to inform PMON of the correct port.) Always
remember to subsequently make your changes in the init file or spfile so that
your changes persist.
However, setting up services on RAC is a little different. On cluster databases
you must never, ever, use the above described method. In other words, do
not set the SERVICE_NAMES parameter directly! Although you cannot use
SERVICE_NAMES here, there are three different tools you can use to set up
services on RAC:
• Command line and Inbuilt PL/SQL functions
• DBCA – The “DataBase Creation Assistant”
• EM Database Control.
The command line is the lowest level interface; both DBCA and EM Database
Control call these commands under the covers. Command line configuration
in RAC has two steps: configuring the clusterware and configuring the
database. The clusterware is first configured through the srvctl command.
Next, the database is set up using the DBMS_SERVICE inbuilt PL/SQL
package. A command line dialogue that creates a service named “reports” is
shown below:
Services aid in resource management. The building blocks of Oracle’s
resource management system are consumer groups. These consumer groups
[oracle@rh4lab15 ~]$ srvctl add service -d db2rac1
are mapped to services, modules, and actions graphically using the > -s reports -r db2rac11,db2rac12 -P basic
Enterprise Manager (EM) Database Control or programmatically through the
DBMS_RESOURCE_MANAGER package. The database engine is then capable SQL> begin
2 dbms_service.create_service(
of assigning execution priorities based on services and can even dynamically 3 service_name=>’reports’,
change the execution priority at runtime as subroutines move from one code 4 network_name=>’reports’,
block to the next. 5 goal=>dbms_service.goal_none,
6 dtp=>FALSE,
Services also aid and enhance job control. First, all of the performance tuning 7 aq_ha_notifications=>FALSE,
and resource management benefits mentioned above also apply to jobs when 8 clb_goal=>dbms_service.clb_goal_long
9 );
they run within a service. You can easily view performance statistics for 10 end;
groups of jobs and control execution priority of jobs by assigning them to 11 /
services. Second, job control is enhanced because job execution is tied to
PL/SQL procedure successfully completed.
the status of its service: In a multi-node RAC environment, the job will only
execute on cluster nodes where the service is enabled. If a service is [oracle@rh4lab15 ~]$ srvctl start service -d db2rac1
completely disabled, then jobs assigned to that service will be suspended > -s reports
until the service is enabled. Note that you must use DBMS_SCHEDULER to [oracle@rh4lab15 ~]$ lsnrctl services
make use of this functionality – the older, deprecated DBMS_JOB system is Service “reports.lab.ardentperf.com” has 2 instance(s)
not integrated with services. Note, also, that you can apply these principles in Instance “db2rac11”, status READY, has 2 handler(s)
Instance “db2rac12”, status READY, has 1 handler(s)
single-instance, non-RAC environments as well.
As we noted, services are a lot more than just a connect method. They add
The command line offers the most flexibility and control; however, some
useful functionality for tuning, resource management and job control to all
DBAs prefer to use a “point-and-click” interface when possible. Both of the
Oracle databases – even non-RAC systems!
other setup methods are “point-and-click.”
2. There Are Four Different Ways to Set up Services In Oracle Database 10g, DBCA can also be used to set up services – DBCA is
There are several ways to set up services, and the procedure varies slightly much more than just a database “creation assistant” – it is also a tool for
depending on your environment. There are four possible ways you can reconfigurations such as adding new cluster nodes and reconfiguring ASM.
achieve this. However, for some strange reason, this functionality has been removed in
Oracle Database 11g. DBCA in Oracle Database 10g also has a service
Page 14 n 3rd Qtr 2008

3. management page that is available in two places: the database creation wizard 3. Service Configuration Is Stored in Four Different Places
and directly from the main menu. DBCA offers the advantage that it Perhaps one of the most surprising and little-known facts about services is
automatically maintains the server’s TNSNAMES.ORA file as well. This is just how spread out the configuration is. Configuration for services is stored
especially useful for PRECONNECT failover configurations that require a in no less than four places: the spfile (or init file), the data dictionary, the
configuration that is not immediately obvious. However, DBCA only performs cluster registry and the filesystem. Naturally the first place that comes to mind
half of the service configuration—it only calls srvctl but does not call the is the initialization file, because that is where the SERVICE_NAMES parameter
DBMS_SERVICE package. In this case, the service will be automatically is stored. But there are, in fact, quite a few additional initialization
created in the database with default values on the first client connect. A parameters that are also related to service configuration. These are listed in
screenshot of DBCA (shown below) is being used to create services. the following Table.
SERVICE_NAMES Should never be set in RAC databases. For single-instance
databases only, set this parameter to create services.
LOCAL_LISTENER PMON needs this parameter to register services with the listener
on the local machine.
REMOTE_LISTENERS PMON needs this parameter to register its services with listeners
on other servers.
DB_DOMAIN This is automatically appended to service names that do not have
a domain.
DISPATCHERS Optionally, dispatchers can be explicitly configured for each service.
STATISTICS_LEVEL Must be TYPICAL or ALL to enable gathering service-related statistics.
As might be expected, the data dictionary contains the bulk of service-related
configuration. Most importantly, it contains a master list of all configured
services and their properties. This includes metadata, load balancing
configuration, notification options, server-side failover configuration and the
DTP (Distributed Transaction Processing) flag. The data dictionary also
The EM Database Control is Oracle’s recommended tool for maintaining contains consumer group mappings, alert thresholds and mapping to job
services. It is just as simple as DBCA, but it allows you to edit a number of classes. The relevant data dictionary views are listed below in the table.
additional service properties. You can specify service level thresholds,
DBA_SERVICES Lists all services registered with the database
consumer groups, job classes, notification properties, load balancing goals (including offline services) and their properties.
and distributed transaction processing options. Database Control calls both It does not include deleted services.
the srvctl program and the DBMS_SERVICE package. However, it does not
DBA_RSRC_MAPPINGS Lists mappings between services and consumer
maintain the TNSNAMES.ORA file, so you will have to write the networking groups. These are used by the resource manager to
entries yourself. prioritize jobs and client connections at run-time.
The fourth and final method is shown below using a screenshot of EM DBA_THRESHOLDS Lists thresholds for Oracle database’s built-in alerting
Database control creating this very same service. system. Includes response-time alerts for services.
DBA_SCHEDULER_JOB_CLASSES Displays mappings between scheduler jobs
and services.
In Oracle cluster databases, services are managed by the clusterware. That is
why the cluster registry (OCR) is absolutely critical even though it only exists
on RAC systems. There are two types of configuration data stored in the OCR:
resources and stringpairs. You should never manually edit these settings.
They can be viewed with the crsstat and ocrdump utilities. The srvctl utility
automatically creates a resource with the appropriate properties and a
number of stringpairs for each service you create. Some of the properties
include the start/stop scripts, dependencies, restart policies, preferred/
available status, enabled/disabled status and TAF policies.
Finally, there is an important bit of configuration stored in the filesystem: the
TNSNAMES.ORA file. On the database server, this file contains the connection
information for LOCAL_LISTENER and REMOTE_LISTENERS. On the clients,
OEM Database Control is generally the preferred method for service
of course, it contains the connect descriptors for connecting to the database
configuration. However, it is good to be aware of the different options, and
with load balancing and client-side failover. Getting this file right is fairly
we have listed all of them in this section.
continued on page 16
3rd Qtr 2008 n Page 15

4. Oracle Services on RAC: Five Things You Might Not Know continued from page 15
straightforward, yet often overlooked. (It’s surprising how many people
mistakenly use the server IP address instead of the VIP to connect!)
4. There Are Multiple Layers of Load Balancing and Connection Failover
Most RAC database administrators know that services are used for load
balancing and connection failover in RAC. However, many of them may not
realize just how many levels of load balancing and connection failover exist
for services.
There are three levels of load balancing in Oracle: client-side connection
load balancing, server-side connection load balancing, and run-time load
balancing. Client-side connection load balancing is the simplest and most
straightforward; the client will randomly choose one of the TNSNAMES
addresses for each new connection, and the connections will be distributed There are also three levels of load balancing in the TNSNAMES and three
roughly the same across the listed connections. Server-side connection load different ways to configure run-time failover. In the TNSNAMES files, both
balancing is enabled by default and is slightly more sophisticated than client- failover and load balancing can be configured in the DESCRIPTION_LIST
side load balancing. The listener knows the current load on each database block, the DESCRIPTION block and the ADDRESS_LIST block. Personally, I
instance and can re-route client connections to the most responsive instance have never yet needed to configure any more than just the ADDRESS_LIST.
or the instance with the smallest number of sessions. Third, Oracle supports
run-time load balancing within JDBC and ODP.NET connection pools. When a
ARDENTACCT.LAB.ARDENTPERF.COM =
thread requests a connection from the pool Oracle can automatically return a (DESCRIPTION_LIST =
connection on the least loaded cluster instance. Which of these three (LOAD_BALANCE = ON)
methods should be used for load balancing? At a minimum, I usually (FAILOVER = ON)
recommend the first two—if not all three! Note that there is an inverse (DESCRIPTION = ... )
(DESCRIPTION =
relationship between the quality of load balancing and the complexity of (LOAD_BALANCE = ON)
setup. This relationship is shown here. (FAILOVER = ON)
(ADDRESS_LIST = ... )
(ADDRESS_LIST =
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab12)(PORT = 1521))
(ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab13)(PORT = 1521))
)
(CONNECT_DATA= (SERVICE_NAME = orcl.lab.ardentperf.com))
)
)
As for Transparent Application Failover (TAF), run-time failover can be
configured with client-side TAF, server-side TAF and Fast Connection Failover
(FCF). So, which of these methods should be used for failover? This is
determined by your environment: FCF is only available for connection
There are also three different levels of connection failover in RAC: the service pools and server-side TAF is only available in Oracle Database 10g and
level, the connection level, and the runtime level. The broadest layer of newer databases.
failover occurs at the service level on the database server. During normal
Load balancing and failover are two of the most important features of
operation a service runs only on its PREFERRED nodes, but if the clusterware
services; they are fundamental to meeting the HA goals of cluster projects.
detects the failure of an instance then it will automatically relocate that
So it is very important that administrators are well acquainted with every level
service to an AVAILABLE node. A second layer of failover occurs when the
of load balancing and failover that services offer.
client makes a connection to the database. Like client-side load balancing it
is easily configured in the TNSNAMES file and causes the client to 5. Services Don’t Always Do What You Would Expect
automatically retry the connection with a different address when the first
address fails. Lastly, Oracle provides run-time failover: Under certain Lastly, it comes as no surprise that services do not always do what you would
conditions, the client can detect a node failure and automatically move expect. We never know exactly how new technology will surprise us, so it
existing connections to a surviving node with no interruption to the seems worth reviewing a few specific points where services have behaved
application. Experienced RAC administrators may notice that the third run- differently than what some database administrators expected.
time failover option seems to come up a lot when talking about load First, a “preferred-available” configuration in a multi-node RAC environment
balancing and failover. Again, there is an inverse relationship between the does not perform the same as a single-node configuration. If a service is
client-oriented and server-oriented connection failover in these three setups. configured to be “preferred” on one node and “available” on a second node,
This is also shown in the next diagram. it does mean that the service only runs on the preferred node during normal
operation. However, the cluster database is fully operational on both nodes
Page 16 n 3rd Qtr 2008

5. and there is an associated overhead (such as maintaining cache coherency
and global locks); hence, the performance profile will not be the same as a
single-node database. If you are clustering for the sole reason of high
availability, then perhaps you should consider whether a cold-failover cluster
would meet your requirements. When configured properly, it is possible to
achieve failover times of fewer than 120 seconds in such cold-failover cluster
environments. Additionally, the license and support fees for failover clusters
are often significantly lower than RAC, and that is an added bonus.
The second unexpected behavior of services is that there is no automatic
failback of services on clusters. If a service fails over to an “available” node,
then the service will not automatically move back to the preferred node when
you restart it. The service must be manually switched back.
The third unexpected behavior is related to the previous “feature”—services
sometimes will not automatically start when the server is started. In Oracle
Database 10g the clusterware always attempts to return services to their last-
known state. If you shut down specific services using srvctl, then the service
will not start after a reboot because the last known status of the service was
“offline.” However, if a node shuts down unexpectedly, then the clusterware
will restart the services on reboot.
Finally, if you are on Oracle Database 10g or earlier, there is a bit of
unexpected behavior related to parallel execution on RAC. Parallel query
slaves inherit the service name for the purposes of reporting performance
statistics, but they will run on all nodes of the cluster, completely ignoring
PREFERRED and AVAILABLE assignments for the service. The workaround is
to use the INSTANCE_GROUPS and PARALLEL_INSTANCE_GROUPS
parameters to limit parallel query execution to the nodes where your service
is running. However, you will need to manually keep these parameters
updated as you move services around. This behavior is fixed in Oracle
Database 11g although the PARALLEL_INSTANCE_GROUPS parameter can still
be used to override the default behavior and run the parallel query slaves on
a different subset of cluster nodes.
Conclusion
After reviewing Oracle services, it is easy to see why they are so important to
modern databases—and especially to cluster database systems. They add a
completely new level of workload and performance management, and they
provide the foundation for RAC’s most important availability features. The
extensive benefits of services far outweigh the shortcomings and small
learning curve for using them. The bottom line is that understanding services
has become a basic, core responsibility of anyone who plans or maintains
Oracle systems today, and that they should be configured for all databases.
n n n About the Author
Jeremy Schneider is an Oracle Database consultant who has worked
with small and large organizations across the country. His interests
include applying grid and cluster technology to large-scale problems
and he spends a lot of time working with RAC databases. Schneider is
also an enthusiastic supporter of user groups and open source projects,
and maintains an Oracle technology blog at www.ardentperf.com/.
3rd Qtr 2008 n Page 17