havcs-410-101 a-2-10-srt-pg_3

VERITAS Cluster Server for
UNIX, Example Application
Configurations
HA-VCS-410-101A-2-10-SRT (100-002147)

COURSE DEVELOPERS
Bilge Gerrits
Siobhan Seeger
Dawn Walker
LEAD SUBJECT MATTER
EXPERTS
Geoff Bergren
Connie Economou
Dave Rogers
Jim Senicka
Pete Toemmes
TECHNICAL
CONTRIBUTORS AND
REVIEWERS
Billie Bachra
Barbara Ceran
Bob Lucas
Gene Henriksen
Disclaimer
The information contained in this publication is subject to
change without notice. VERITAS Software Corporation
makes no warranty of any kind with regard to this guide,
including, but not limited to, the implied warranties of
merchantability and fitness for a particular purpose.
VERITAS Software Corporation shall not be liable for errors
contained herein or for incidental or consequential damages in
connection with the furnishing, performance, or use of this
manual.
Copyright
Copyright © 2005 VERITAS Software Corporation. All
rights reserved. No part of the contents of this training
material may be reproduced in any form or by any means or
be used for the purposes of training or education without the
written permission of VERITAS Software Corporation.
Trademark Notice
VERITAS, the VERITAS logo, and VERITAS FirstWatch,
VERITAS Cluster Server, VERITAS File System, VERITAS
Volume Manager, VERITAS NetBackup, and VERITAS
HSM are registered trademarks of VERITAS Software
Corporation. Other product names mentioned herein may be
trademarks and/or registered trademarks of their respective
companies.
VERITAS Cluster Server for UNIX, Example Application
Configurations
April 2005 Release
VERITAS Software Corporation
350 Ellis Street
Mountain View, CA 94043
Phone 650–527–8000
www.veritas.com

Table of Contents i
Copyright © 2005 VERITAS Software Corporation. All rights reserved.
Course Introduction
VERITAS Cluster Server Curriculum ................................................................ Intro-2
Guide Prerequisites........................................................................................... Intro-3
Guide Objectives............................................................................................... Intro-4
Legend .............................................................................................................. Intro-6
Example 1: Clustering Applications
Application Service Overview.................................................................................. 1-3
VCS Agents for Managing Applications .................................................................. 1-6
The Application Agent........................................................................................... 1-11
Lab 1: Creating an Application Resource.............................................................. 1-15
Example 2: Clustering Databases
Database Service Example..................................................................................... 2-3
VCS Database Agents ............................................................................................ 2-5
Database Preparation ............................................................................................. 2-9
The Enterprise Agent for Oracle ........................................................................... 2-15
Database Failover Behavior.................................................................................. 2-30
Additional Oracle Agent Functions........................................................................ 2-32
Lab 2: Creating Oracle Resources........................................................................ 2-38
Example 3: Clustering NFS
NFS Service Overview............................................................................................ 3-3
Preparing NFS for High Availability......................................................................... 3-5
Testing the NFS Service ....................................................................................... 3-12
Configuring an NFS Service Group....................................................................... 3-16
NFS Lock Failover in VCS 4.1 .............................................................................. 3-19
Lab 3: Creating NFS Resources ........................................................................... 3-29
Appendix A: Lab Synopses
Lab 1 Synopsis: Creating an Application Resource................................................ A-2
Lab 2 Synopsis: Creating Oracle Resources .......................................................... A-6
Lab 3 Synopsis: Creating NFS Resources............................................................ A-16
Appendix B: Lab Details
Lab 1 Details: Creating an Application Resource.................................................... B-3
Lab 2 Details: Creating Oracle Resources............................................................ B-11
Lab 3 Details: Creating NFS Resources ............................................................... B-31
Appendix C: Lab Solutions
Lab 1 Solutions: Creating an Application Resource................................................ C-3
Lab 2 Solutions: Creating Oracle Resources........................................................ C-11
Lab 3 Solutions: Creating NFS Resources ........................................................... C-35
Appendix D: Design Worksheet: Template
Table of Contents

ii VERITAS Cluster Server for UNIX, Example Application Configurations

Intro–2 VERITAS Cluster Server for UNIX, Example Application Configurations
VERITAS Cluster Server Curriculum
The VERITAS Cluster Server curriculum is a collection of courses that are
designed to provide a full range of expertise with VERITAS Cluster Server (VCS)
high availability solutions.
VERITAS Cluster Server for UNIX Fundamentals
This course covers installation and configuration of common VCS configurations,
focusing on two-node clusters running application and database services.
High Availability Fundamentals
This course covers VERITAS Volume Manager and VERITAS Cluster Server
Fundamentals.
VERITAS Cluster Server for UNIX, Example Application
Configurations
This guide provides optional supplementary material for VERITAS Cluster Server
Fundamentals and High Availability Fundamentals. The instructor may opt to
present some of this material in class, depending on time constraints and student
interest.
VERITAS Cluster Server Curriculum
Courses
High Availability
Fundamentals
VERITAS Cluster
Server, Example
Application
Configurations
VERITAS
Cluster Server
Fundamentals
Optional
Guide

Introduction Intro–3
Guide Prerequisites
This guide assumes that you have mastered the fundamentals of VERITAS Cluster
Server. You should also have an administrator-level understanding of one or more
UNIX platforms, including the ability to configure systems, storage devices, and
networking in multiserver environments. A basic understanding of VERITAS
Volume Manager is also required.
Guide Prerequisites
To successfully use the material in this
guide, you should have the following
expertise:
VERITAS Cluster Server fundamentals
VERITAS Volume Manager basic
configuration
System and network device configuration

Guide Objectives
The VERITAS Cluster Server for UNIX, Example Application Configurations
guide simulates the job tasks you perform to configure common applications in a
VCS environment.
Guide Objectives
After completing the VERITAS Cluster Server for
UNIX, Example Application Configurations
guide, you will be able to:
Configure an Application resource to manage an
application in a VCS environment.
Configure Oracle and Netlsnr resources and set
up detail monitoring of an Oracle database
instance.
Configure NFS and Share resources to make an
NFS service highly available.

Guide Resources
This guide contains modules with conceptual and configuration information for
each topic. Lab exercises enable you to practice your new skills, and solutions are
provided to guide you through the exercises.
Lab materials are provided in three forms, with increasing levels of detail to suit a
range of student expertise levels.
• Appendix A: Lab Synopses has high-level task descriptions and design
worksheets.
• Appendix B: Lab Details includes the lab procedures and detailed steps.
• Appendix C: Lab Solutions includes the lab procedures and steps with the
corresponding command lines required to perform each step.
• Appendix D: Design Worksheet Template provides a blank design
worksheet.
Additional supplements may be used in the classroom or provided to you by your
instructor.
Guide Resources
Participant Guide
– Modules
– Appendix A: Lab Synopses
– Appendix B: Lab Details
– Appendix C: Lab Solutions
– Appendix D: Design Worksheet Template
Supplements
– VCS Simulator: van.veritas.com
– Tips & Tricks: www.veritas.com/education

Legend
These are common symbols used in this guide.
Symbol Description
Server, node, or cluster system (terms
used interchangeably)
Server or cluster system that has faulted
Storage
Application service
Cluster interconnect
Wide area network (WAN) cloud

Client systems on a network
VCS service group
Offline service group
VCS resource
Symbol Description

Example 1
Clustering Applications

1–2 VERITAS Cluster Server for UNIX, Example Application Configurations
Introduction
Overview
This module describes how agents manage resources in a VCS cluster. In this
module you are also introduced to the VCS bundled Application agent.
Importance
Understanding how agents work helps you to determine the best way to manage
your application service in a high availability environment.
Outline of Topics
• Application Service Overview
• VCS Agents for Managing Applications
• The Application Agent
Configure the Application agent.The Application Agent
Describe how agents manage
applications in a VCS environment.
VCS Agents for Managing
Applications
Describe how application services
work in an HA environment.
Application Service
Overview
After completing this lesson, you
will be able to:
Topic
Lesson Topics and Objectives

Example 1 Clustering Applications 1–3
1
Application Service Overview
Recall that application services consist of three basic building blocks or
components:
• Application—the processes or daemons that handle client requests
• Storage—data storage resources
• Network—resources that enable clients to access the service from the network
In order to make an application service highly available using VCS, you configure
the applicable resources corresponding to the objects that are logically contained
within these components.
Lessons in the VERITAS Cluster Server Fundamentals guide focused on
configuring the storage and network components in VCS. This module
concentrates on the VCS agents you can use to manage the application component.
Note: Before bringing VCS into the environment, ensure that all components are
properly configured, as described in the “Preparing Services for High Availability”
lesson in the VERITAS Cluster Server Fundamentals Participant Guide.
Application Service Overview
NetworkNetwork
StorageStorage
ApplicationApplication
S1
S2
Daemons
ProcessesFile System
Volume
Disk Group
IP Address
NIC

Application Requirements
The most important requirements for an application to run in a cluster are crash
tolerance and host independence. This means that the application should be able to
recover after a crash to a known state, in a predictable and reasonable time, on two
or more hosts.
Most commercial applications today satisfy this requirement. More specifically, an
application is considered well-behaved and can be controlled using a VCS agent if
the application meets the following criteria:
• There must be a defined start and stop procedure that can be automated.
• If the application fails, it must be possible to clean up anything left behind,
such as defunct processes, reserved memory segments, and so on.
• It should be possible to check the status of the application using an automated
method. Monitoring an application can be performed in many ways. In general,
the closer a test emulates what the users of the application do, the better the test
is in discovering problems. However, this may impact the performance of the
application. VCS may also detect problems that you want administrator
intervention to solve, such as running out of tablespace in a database.
Therefore, the monitoring should be carefully balanced to ensure that the
application is up while minimizing the overhead and limiting VCS to
appropriate monitoring actions.
Application Requirements
In order to control an application using a VCS
agent, the application must:
Have a defined start and stop procedure.
Provide the ability to clean up after operational failures.
Provide the ability to monitor the health of the application
using an automated method.
Provide the ability to store its data to external shared
disks.
Be crash-tolerant; that is, it must be capable of being
restarted to a known state after sudden failures.
Be able to run on all servers designated as potential
hosts, and must have:
– No license issues tied to individual servers
– No hostname dependencies

1
• If the application has any data that is updated during operation, the data should
be stored on an external shared disk. This also means that the application
should not keep any data in memory. Because the application is started on
another host after operational failures, any data that other hosts cannot reach is
lost. This precludes the use of any NVRAM accelerator boards or other disk-
caching mechanisms contained in a local host. Disk caching for performance
should only be performed on the external array.
• The application must be able to restart to a known state without manual
intervention after operational failures, such as a system failure. For example,
an Oracle database guarantees this by acknowledging the clients only after
writing the changes to a log file on the shared disk (the redo logs) when a
commit changes request is received. Anything that has not been committed or
acknowledged is the responsibility of the client and is kept there. So when the
system fails, the database can be brought up on another system because the
other system has access to the redo logs and can determine what has been
committed and written to disk. The database can be brought to the exact point
of time of the crash by applying these redo logs.
• The application must be host-independent. Some applications are licensed to
the host name, which means that the application does not start on another host
unless the host name is changed. A highly available application should not
have any features that tie it to a specific system in the cluster.

VCS Agents for Managing Applications
Agents are programs—combinations of binaries, libraries, scripts—that function
as the intermediary between resources and the VCS engine. Every resource type
has an agent that interacts with all resources of that type on behalf of VCS.
You have several options for selecting an agent to manage an application.
• You can purchase VERITAS Enterprise agent packages as add-ons to the basic
VERITAS Cluster Server product. Some advantages of using Enterprise agents
are that they are:
– Easy to install and configure
– Fully tested and supported for the documented versions of the operating
system and application
– Enabled for detail monitoring (most agents) to perform additional
verification of the application status
• You can use the Application or Process agents that are bundled with the base
VCS product. Depending on your application and requirements, you may need
to write custom code or scripts when using the Application agent. For example,
if you want to change the way that the agent monitors your application, you
can create your own script and configure the agent to use that script.
• VERITAS has created partnerships with many application vendors to enable
them to create agents for managing their software.
• You can modify an existing agent or create a new agent if you have specialized
requirements for managing your application in a high availability environment.
VERITAS provides separate documentation and training to enable you to
develop VCS agents.
VCS Agents for Managing Applications
You can purchase Enterprise agents for popular
applications. Examples include:
– Oracle, DB2, Informix, Sybase
– Sun ONE, Apache, SAP
You can also manage some applications using bundled
agents:
– Application
– Process
Some software vendors provide VERITAS-Enabled agents
for their applications, such as:
MySQL, NCR, and webMethods
Customized agents can be developed in-house or by
VERITAS Consulting Services.
Oracle
Application
Sun ONE
Process
DB2

1
Key Properties of VCS Agents
Key properties of VCS agents include:
• Only one agent process runs on a system for each configured resource type.
That agent manages all resources of that type on that system.
• An agent runs a single operation on a resource at one time. This ensures that
agents on different systems cannot be affecting a resource simultaneously on
different systems.
• Agents are multithreaded so that operations can be carried out in parallel on
multiple resources of the same type. For example, the IP agent can be bringing
the IP1 resource online at the same time it is monitoring the IP2 resource.
• A resource cannot be managed without an agent.
• If there are no resources of a particular type anywhere in the cluster, the agent
for that type is not started.
VCS agents are located within subdirectories of the VCS bin directory,
/opt/VRTSvcs/bin.
Key Properties of VCS Agents
Only one agent daemon runs on a system for each
configured resource type.
An agent runs a single operation on a resource at one
time.
Agents are multithreaded so that operations can be
carried out in parallel on multiple resources of the
same type.
A resource cannot be managed without an agent.
If there are no resources of a particular type
anywhere in the cluster, the agent for that type is not
started.

VCS Agent Functions
Most VCS agents perform this common set of tasks:
• Bring resources online when requested by the VCS engine.
• Take resources offline upon request.
• Periodically monitor resources and send status information to the VCS engine.
• Clean up if a resource fails to go offline properly.
Agents also:
• Restart resources when they fault (depending on the resource configuration).
• Send a message to the VCS engine and the agent log when errors are detected.
Some agents support multiple levels of monitoring. For example, most database
agents can be configured to perform two levels of monitoring:
• Basic monitoring: Scans the process table for database processes
• Detail monitoring: Performs a transaction on a test table in the database
Agents may also perform other functions, depending on the type of resource they
manage. The documentation for each agent describes which functions each one
supports. The VERITAS Cluster Server Agent Developer’s Guide provides a
comprehensive description of each supported agent function.
For information about resource states and transitions, see the “Job Aid” appendix
in this Participant Guide.
VCS Agent Functions
To manage an application, the VCS agent must be
able to:
Start a specified program.
Stop the specified program.
Monitor the program.
Some agents have multilevel monitoring functions
that can be configured.
– Basic-level monitoring may only check the process
table for applicable processes.
– Second-level monitoring, if enabled, can execute
code, simulating client access to the application.
Clean up after a fault.

1
The VCS Agent Framework
The modular design of VCS provides an agent framework to enable additional
agents to be added without affecting the core engine functions. Agents manage
resources by calling the appropriate function or entry point, and HAD manages the
agents.
When an event occurs and HAD determines that an action must be taken, it calls
the appropriate agent. The agent then invokes the script or function for the
applicable entry point, passing a set of input parameters as arguments to perform
the resource management task requested. The scripts called by the agent are
referred to as entry points into the agent.
Entry points that are common to most agents are:
• Online: Starts or creates the resource
• Offline: Stops or removes a resource
• Monitor: Determines resource status
• Clean: Forcibly takes a resource offline
Entry points are determined by the type of resource that an agent is managing. For
example, a NIC resource is persistent, so it does not have online and offline entry
points. Online-only resources, such as NFS, do not have an offline entry point.
However, all agents are required to have monitor and clean entry points.
In most cases, entry points are executed as well-known UNIX shell commands and
are easily customizable for developers experienced with creating shell scripts.
The VCS Agent Framework
Agents consist of an agent binary file plus a
collection of scripts or C++ functions that
correspond to each agent function.
The agent functions are referred to as entry
points.
Most agents include the online, offline, monitor,
and clean entry points.
Entry points are often implemented as Perl or
shell scripts, in which case they can easily be
customized.
Agents can also be implemented as compiled
executable files.

How Agents Work
When the VCS engine requires an action to be taken for a resource—as the result
of an administrative command or in response to an event, such as a fault—HAD
calls an agent with a set of parameters. The agent takes a set of parameters as input
from HAD. These parameters are the values specified by resource attributes in the
main.cf file. The order in which they are passed is specified by the ArgList
attribute of the corresponding resource type defined in types.cf.
The agent calls the entry point with the name of the resource followed by
arguments—values of the resource attributes (main.cf)—in the order specified
in the ArgList attribute for the resource type (types.cf).
Note: The diagram shows a conceptual view of the communication and
relationships among VCS components. The agent does not read the configuration
files directly. The VCS engine has the configuration in memory and passes the
configuration information to the agent when it starts or a new resource is created.
The agent then stores the configuration information in memory.
HAD
How Agents Work
Online
Entry Point
$1 AppVolume
$2 AppVol
$3 AppDG
Volume AppVolume (
Volume = AppVol
DiskGroup = AppDG
)
main.cfmain.cf
Online AppVolume
vxrecover -g AppDG –bs AppVol
Volume Agent
online
offline
monitor
clean
type Volume (
static str ArgList[] =
{ Volume, DiskGroup }
. . .
)
types.cftypes.cf App
Volume
In-memory configuration

1
The Application Agent
If you have an application that does not have an available Enterprise agent, you
may be able to use the Application agent to make the application highly available.
The Application agent can be used to control more complex applications and can
be easily customized. Whereas the Process agent can be used to monitor a single
process, the Application agent can manage applications consisting of multiple
processes.
Entry Points
• Online: Runs StartProgram with the specified parameters in the specified user
context
• Offline: Runs StopProgram with the specified parameters in the specified user
context
• Monitor: If no MonitorProgram is specified, verifies that all processes
specified in PidFiles and MonitorProcesses are running
If MonitorProgram is defined, the agent executes the MonitorProgram.
• Clean: Kills all the processes specified in PidFiles or in MonitorProcesses
If CleanProgram is defined, the agent executes the CleanProgram.
The Application Agent
The Application agent:
Simplifies managing applications with multiple
processes and enables detailed monitoring.
Requires no additional licensing; is bundled with
VCS.
Is easily customized by creating scripts that the
agent uses within its entry points.
Manages applications using these entry points:
Online Brings an application online using StartProgram
Offline Takes an application offline using StopProgram
Monitor Monitors the status of the application
Clean Takes the application offline using CleanProgram
or kills all the processes specified for the
application

Application Resource Definition
The required attributes listed in the slide must be defined to enable the Application
agent to manage a resource.
The optional attributes, CleanProgram and User, can be configured to provide
additional control of the application.
Application Resource Definition
Required Attributes:
StartProgram Name of executable to start application
StopProgram Name of executable to stop application
Plus one or more of the following:
MonitorProgram Name of executable to monitor application
MonitorProcesses List of processes to be monitored
PidFiles List of files that contain the process ID of
the processes to be monitored
Optional Attributes:
CleanProgram Executable file that forcibly kills the
application and performs cleanup tasks
User UNIX account used as the execute
environment for entry points
The default is root.

1
Application Resource Configuration
The example in the slide shows how an Application resource is configured to make
the Samba file sharing application highly available.
Note: The MonitorProcess values must match the output displayed by the ps
command exactly. For example, if the processes are displayed with full path
names, you must include the full path name when specifying the processes to
monitor.
Application Resource Configuration
Configuration prerequisites:
– The application should have its own start and stop
programs.
– It must be possible to monitor the application by either
running a program that returns 0 for failure and 1 for
success or by checking a list of processes.
Sample configuration:
Application samba_app (
StartProgram = “/usr/sbin/samba start”
StopProgram = “/usr/sbin/samba stop”
PidFiles = { “/var/lock/samba/smbd.pid” }
MonitorProcesses = { “smbd nmbd” }
)

Summary
This lesson described how agents manage resources in a VCS cluster. In this lesson
you are also introduced to the VCS bundled Application agent.
Next Steps
Now that you understand how agents manage applications, you can configure
other common types of high availability services.
Additional Resources
• VERITAS Cluster Server Bundled Agents Reference Guide
This guide describes each bundled agent in detail.
• VERITAS Cluster Server Agent Developer’s Guide
This guide provides detailed information about the agent framework and
creating or customizing agents.
• VERITAS Cluster Server Agent Development Course
This course provides detailed information about the agent framework and
creating or customizing agents. The course includes labs to customize existing
agents and create new agents.
• VERITAS Cluster Server Fundamentals Course
This course provides detailed information about configuring VCS service
groups and resources.
• http://www.veritas.com
A list of agents available from VERITAS and partners is available by selecting
Products—>Cluster Server—>Agents and Options link.
Lesson Summary
Key Points
– VCS provides a variety of agents to manage all
types of applications.
– The Application agent is easily configurable for
basic management of an application.
Reference Materials
– VERITAS Cluster Server Bundled Agents
Reference Guide
– VERITAS Cluster Server Agent Developer's Guide
– VERITAS Cluster Server Agent Development
Course
– VERITAS Cluster Server Fundamentals Course
– http://www.veritas.com

1
Lab 1: Creating an Application Resource
Labs and solutions for this lesson are located on the following pages.
Appendix A provides brief lab instructions for experienced students.
• “Lab 1 Synopsis: Creating an Application Resource,” page A-2
Appendix B provides step-by-step lab instructions.
• “Lab 1 Details: Creating an Application Resource,” page B-3
Appendix C provides complete lab instructions and solutions.
• “Lab 1 Solutions: Creating an Application Resource,” page C-3
Goal
The purpose of this lab is to configure the Application agent to manage a service
with multiple processes.
Prerequisites
Obtain any classroom-specific values needed for your classroom lab environment
and record these values in your design worksheet included with the lab exercise
instructions.
Results
Each student’s service can be started, monitored, and stopped on each cluster
system.
Lab 1: Creating an Application Resource
nameApp1
name
Mount1
nameVol1
nameDG1
nameProxy1
nameIP1Delete the Process
resource and add
an Application
resource.
Delete the Process
resource and add
an Application
resource.
nameSG1nameSG1
See the next slide for classroom values.See the next slide for classroom values.

Example 2
Clustering Databases

Introduction
Overview
This module describes how to manage database services in a VCS environment.
An Oracle service group is used as an example showing specific agents and
resource types that are provided with VCS enterprise agent for Oracle.
Importance
Database services are a key component of most high availability environments.
Understanding how you can use VCS to manage your database software is critical
to implementing a complete high availability solution.
Outline of Topics
• Database Service Example
• VCS Database Agents
• Database Preparation
• The Enterprise Agent for Oracle
• Database Failover Behavior
• Additional Oracle Agent Functions
Describe common database service
examples.
Database Service Example
Describe VCS agents for managing
databases.
VCS Database Agents
Prepare a database for use in a VCS
environment.
Database Preparation
Implement additional functionality
provided by the Oracle agent.
Additional Oracle Agent
Functions
Configure database failover behavior.Database Failover
Behavior
Install and configure the VCS Oracle
agent.
The Enterprise Agent for
Oracle
will be able to:
Topic

Example 2 Clustering Databases 2–3
2
Database Service Example
As with most application services, database services generally include three basic
components:
• Application—the processes providing the service
The diagram shows a simple example of a database service.
Generic Database Service Example
File System
/db
Contains data
and log files
File System
/db
Contains data
and log files
Database/
Listener
Database/
Listener
IP Address
10.10.21.54
IP Address
10.10.21.54
VxVM Volume
dbvol
VxVM Volume
dbvol
Network Port
NIC_device
Network Port
NIC_device
VxVM Disk
Group dbdg
VxVM Disk
Group dbdg
Network
Component
Application
Component
Storage
Component

VCS manages database services in the same way it manages any other application
service. Resources are contained within the service group for each storage and
network and dependencies are defined to ensure that VCS brings the supporting
resources online before attempting to start the database.
The type of resource you use to manage the database depends on your database
software and your specific management requirements.
Corresponding Database Service Group
/db/db
Database/ListenerDatabase/Listener
10.10.21.5410.10.21.54
dbvoldbvol NIC_deviceNIC_device
dbdgdbdg
DB/Lsnr
Mount
Volume
Disk
Group
NIC
IP

2
VCS Database Agents
You have several options for selecting an agent to manage the database
application.
• You can purchase VERITAS enterprise agent packages as add-ons to the basic
VERITAS Cluster Server product. The advantages of using an enterprise agent
are that it is:
– Easy to install and configure
– Fully tested and supported for the documented versions of the operating
system and database application
Enterprise agents are the primary focus of this lesson.
• You can use the Application agent that is bundled with the base VCS product.
Depending on your database and your requirements, you may need to write
custom code or scripts. For example, if you want to change the way the agent
monitors your database, you modify the monitor entry point script.
• You can modify an existing agent or create a new agent if you have specialized
requirements for managing your database in a high availability environment.
VERITAS provides separate documentation and training to enable you to
develop VCS agents.
Agents for Managing Databases
You can purchase Enterprise agents for:
– Oracle
– DB2
– Informix
– Sybase
– MySQL
You can also manage some databases using the
bundled Application agent or customized
agents.
Oracle
DB2
Informix
Sybase
MySQL
Note: Not all agents are supported on all platforms.Note: Not all agents are supported on all platforms.

Agent Functions
All agents perform the same basic functions—start, stop, monitor, clean. The
enterprise agents for databases each perform similar functions.
The agent starts an instance using the database-specific utility (such as sqlplus
for Oracle), usually with a startup profile file name. The instance is likewise
stopped with the database shutdown utility.
Two levels of monitoring can be configured:
• Basic monitoring: Scans the process table for database daemon processes
• Detail monitoring: Performs a transaction on a test table in the database
Primary Monitoring
The instance can be monitored by scanning the process table for the process IDs
(PIDs) for critical database processes. The processes monitored vary by database.
For example, the Oracle agent monitors the ora_smon, ora_dbw, ora_pmon,
and ora_lgwr processes.
Agent Functions
To manage a database, the VCS agent must be
able to:
Start a specified database instance.
Stop the specified instance.
Monitor the instance.
– Basic-level monitoring may check the process
table for applicable database processes.
– Detail monitoring may write to a test table in the
specified database.
Clean up after a database fault to prevent possible
concurrency violations to enable failover.

2
Detail Monitoring
When detail monitoring is enabled, the monitor script attempts to write to a test
table in a specified database. If this script fails and failover is enabled, the entire
service group fails over to the next available system.
Writing to the database ensures that failures, such as full logs, are detected.
Reading from the database does not detect this type of error.
In order to write to a database table, you must create the test table in the database
that you are monitoring. Then, provide the agent with the access and configuration
information needed to write to the table:
• The database user account with update privileges
• The password for the database user account
• The name of the database table
• SQL statements used to write to the table
Note: Detail monitoring may detect errors that cannot be handled by failover or
restarts, such as a database running out of tablespace. Some agents, such as the
Enterprise Agent for Oracle, provide optional methods of handling these types of
errors. These additional error-handling methods are described later in this lesson.
Detail Monitoring
Agents require this information to access and
write to the table:
A database user account with update privileges
The password for the database user account
The name of the database table
The name of the file containing the SQL
statements used to write to the table

Monitoring Listener Processes
Some database products, such as DB2, Sybase, and Informix, have the database
network connection mechanism built into the database processes and do not
require additional agents.
The Enterprise Agent for Oracle product contains an additional agent for
managing the Oracle Net Services (ONS) listener process for Oracle, which
controls client network connections to Oracle databases.
The Netlsnr agent starts and stops the Oracle Net Services listener process using
the lsnrctl command. Two levels of monitoring can be configured:
• Primary monitoring: Scans the process table for the tnslsnr process
• Secondary monitoring: Uses the lsnrctl command to test the listener
process
Monitoring Listener Processes
Some databases have listeners built-in and do not require
additional VCS agents for monitoring.
The Enterprise agent for Oracle includes an agent to
monitor the listener process, named Netlsnr, which
provides two levels of monitoring:
– Primary: Scans the process table for the tnslsnr
process
– Secondary: Uses lsnrctl to test the listener
Client
Data
Server
Net8
Oracle
Server
Listener
Protocol
ONSUser

2
Database Preparation
Verifying Software Compatibility
Ensure that each system that can host the database instance meets the operating
system and database requirements specified in the agent documentation.
Also, ensure that each system has adequate resources, such as shared memory.
Each system in the SystemList of the service group must be properly configured as
a database server.
Verifying Software Compatibility
Use the latest database agent available for the VCS
version in use.
Verify that the agent supports the version of the
database used in the cluster by checking the agent
release notes.
Examples:
Oracle on SolarisOracle on Solaris
Oracle Agent 4.1
VCS 4.1
Solaris 8, 9, 10
Oracle 8.0.x, 8i (8.1.x), 9i,
10g
Oracle Agent 4.1
VCS 4.1
Solaris 8, 9, 10
Oracle 8.0.x, 8i (8.1.x), 9i,
10g
DB2 on AIXDB2 on AIX
DB2 Agent 4.0
VCS 4.0
AIX 5.2
DB2 ESE 7.2, 8.1
DB2 Agent 4.0
VCS 4.0
AIX 5.2
DB2 ESE 7.2, 8.1

Database Program (Executable) Files
The database binaries can be configured either on the local disks or on the shared
storage. Select the location based on your high availability requirements.
• Configuring the binaries on the local disks may enable you to minimize
downtime due to maintenance. You can update the database software on one
system while the database is running on another.
• Installing binaries on a shared storage disk simplifies configuration, especially
if there are many systems that can run the same database instance and
participate in the service group. If you have many systems, applying updates
on each system can be a significant burden.
Note: Some databases, such as Oracle, install updates in a new directory. This
directory can be on shared storage, which provides a way to use the advantages
of updating one system while the database is running on the other. If binaries
are storage on local disks, you have to update each individual system.
Database Program (Executable) Files
The database program files can be configured either
on the local disks or on shared storage.
Local disk:
– Provides redundancy; offers additional protection in
case the binaries are damaged
– The database must be installed and configured
identically on each system that can run the database
service group.
Shared storage:
– All systems must have the same mount point directory
for the database home directory.
– Simplifies configuration, especially if there are many
systems that can run the same database instance

2
Data Files
You must locate the data files on shared storage so that each failover target system
can access them. Use these guidelines for locating data files in a VCS
environment:
• If using file system-based data files, the file systems that contain these files
must be located on shared disks. Create the same file system mount point on
each system in the cluster.
• If using raw devices, such as Volume Manager volumes, set the permissions for
the volumes to be owned permanently by the database account. For example:
vxedit –g dgname set group=dba user=dba mode=660
volume_name
Note: Volume Manager overrides UNIX permissions. It is not sufficient to
change permissions in UNIX because Volume Manager sets volume
permissions when importing a volume (with default owner is root). You must
also change the Volume Manager permissions, as shown in the slide.
Data Files
If using file system-based data files, the file systems
that contain these files must be located on shared
disks.
Create the same file system mount point on each
system in the cluster.
If using raw devices, such as Volume Manager
volumes, set the permissions for the volumes to be
owned permanently by the database. For example:
vxedit –g dgname set group=dba user=dba mode=660 vol
Note: You cannot change permission only in UNIX.
VxVM sets volume permissions upon import.

Database-Related System Files
Although it is not necessary for the system files listed on the slide to be exactly the
same on all systems that participate in the service group, you must ensure that any
database-related configurations are the same or consistent on all of the systems.
Note: Changes to /etc/system require a reboot to take effect.
Solaris
/etc/passwd
/etc/shadow
/etc/group
/etc/services
/etc/system
/etc/hosts
AIX
/etc/passwd
/etc/security/passwd
/etc/group
/etc/services
/usr/samples/kernel/vmtune
/etc/hosts
Database-Related System Files
The following database-related system files must be
consistent on all systems listed in the SystemList
attribute for the service group:
Database user accounts
Passwords for database user accounts
Database group entries
Listener entries
Database shared memory settings
The database service group (virtual) IP address
HP-UXAIX LinuxSolaris

2
HP-UX
/etc/passwd
/etc/group
/etc/services
/stand/system
/etc/hosts
Linux
/etc/passwd
/etc/shadow
/etc/group
/etc/services
Shared memory settings:
– For drivers built into the kernel, append parameters to the kernel command
line using the boot loader.
– For kernel modules, use /etc/modules.conf.
– For tunable parameters, use sysctl and /etc/sysctl.conf.
/etc/hosts

Network Configuration
Each database service group requires at least one IP address for client connections.
This IP address should fail over together with the database in case of any major
faults.
Therefore, you need to use an IP resource (or an IPMultiNIC resource) and
configure the host name of the service group IP address in the database. The clients
should connect to the host name corresponding to this virtual IP address and not to
the local host names of the servers.
Network Configuration
For listener service failover, include at least one IP
resource in the service group for the database.
Use the host name for this service group IP address
for the host field in the listener configuration file.
LISTENER=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=salesDB)(PORT=1521)
)
(ADDRESS=(PROTOCOL=ipc)(KEY=extproc))))
. . .
listener.oralistener.ora
. . .
10.10.21.54 salesDB
. . .
/etc/hosts/etc/hosts
10.10.21.54
IP

2
The Enterprise Agent for Oracle
Installing Agents
The enterprise agents are provided as a standard software package that can be
installed using the VERITAS Common Product Installation Utility or using the
platform-specific standard installation utility.
Install the agent on each system using this basic installation procedure:
1 Log on as root.
2 Mount the installation CD.
3 Change to the directory containing the agent software packages.
4 Unzip and extract the files from the tar archive.
5 Type the platform-specific command to install the agent.
For example, to install the Oracle enterprise agent, type:
Solaris
pkgadd –d . VRTSvcsor
AIX
installp -ac -d /cdrom/VRTSvcsor.rte.bff VRTSvcsor.rte
HP-UX
swinstall -s /cdrom/oracle_agent/pkgs VRTSvcsor
Linux
rpm -ihv VRTSvcsor-2.0-Linux.i386.rpm
Installing Agents
Use the VERITAS VPI or the platform-specific
installation utility to install the VRTSvcs**
package:
– Solaris: pkgadd
– AIX: installp
– HP-UX: swinstall
– Linux: rpm
Install the agents on each system in the cluster
that can run the database service group.
Solaris HP-UXAIX Linux
Sample Oracle
Agent Guides
Sample Oracle
Agent Guides

Adding Resource Type Definitions
Before you can create database resources, you must add the resource type
definitions to the VCS configuration.
You can add resource types while VCS is running or take VCS offline and modify
the configuration files.
Adding Resource Types to a Running Configuration
These methods are provided for adding resource types while VCS is running:
• Using Cluster Manager:
1 Select File—>Import Types.
2 Browse to select the file containing the type definition.
• Use hatype to add an include statement to the main.cf file that references
the databasetypes.cf file. You can also use hatype commands to add
and modify resource type definitions in types.cf.
• Create and run a command file containing hatype commands.
Adding Resource Types to an Offline Configuration
A simple method for adding resource types requires you to stop VCS, edit the
main.cf or types.cf file, and restart VCS. This is described in detail in the
“Offline Configuration of Service Groups” lesson of the VERITAS Cluster Server
Fundamentals course.
Adding Resource Type Definitions
You can use an online or offline method to add
resource type definitions.
To add types while VCS is running, use Cluster
Manager. Select File—>Import Types.
To add types offline:
1. Copy the databaseTypes.cf file to the
/etc/VRTSvcs/conf/config directory.
2. Use the offline configuration process described
earlier in the VERITAS Cluster Server Fundamentals
course to add this line to main.cf and restart the
cluster using the modified configuration files.
include “databaseTypes.cf”

2
High Availability Database Configuration Overview
Before you begin configuring VCS to manage your database services, prepare your
environment, including equipment, software, and staff.
Coordinate with the database and systems administrators responsible for the
database servers you are adding to the cluster environment. After database services
are placed under control of VCS, the databases cannot be managed separately,
outside of VCS.
As with any high availability service, configure and test each component on the
startup system and each failover system before you perform any VCS
configuration of the service.
This procedure applies to configurations where the database binaries are installed
locally on each system, which reflects the cluster design decisions assumed for this
course. Examples of other configurations are shown later in the lesson. The High
Availability Design Using VERITAS Cluster Server course provides a complete
discussion of design considerations for VCS environments.
High Availability Database Configuration
Overview
Install database.Install database.
Configure database.Configure database.
Test database.Test database.
Stop resources.Stop resources.
Y
N
More
Systems?
More
Systems?
Add resource types.Add resource types.
Add service group.Add service group.
Prepare OS.Prepare OS.
Install agent.Install agent.
Test service group.Test service group.
Add/test resources.Add/test resources.
Y
Binaries on Local StorageBinaries on Local Storage

Oracle Service Group Configuration Example
A service group example for an Oracle database instance contains these types of
resources:
• Netlsnr: Monitors the listener process
• Oracle: Monitors the database and processes
• Mount: Monitors one or more file systems containing:
– Oracle application files (if installed on shared storage)
– Oracle data files
– Oracle log files
• Volume: Monitors one or more volumes for the file systems
• DiskGroup: Monitors one or more disk groups for the volumes
• IP: Monitors the service group IP for the listener process
• NIC: Monitors one or more network interface cards for remote client
connection
The example shown on the slide assumes that the Oracle binaries are located on
local storage. The data files are located on a file system (rather than raw volumes).
The clients access Oracle services using the service group IP address defined by
the IP resource.
Configuration Example: Oracle Service Group
Disk
Group
Volume
Mount
Oracle
Netlsnr
NIC
IP

2
Configuring an Oracle Resource
The required attributes for an Oracle resource are shown in the example excerpt of
the main.cf file.
Note: In version 4.1 of the Oracle agent, Pfile is not a required attribute. However,
Pfile is required for versions prior to 4.1.
Agent Entry Points
• Online: Uses the sqldba/svrmgrl/sqlplus startup -f $Pfile
command to start the database
The $Pfile argument is the startup profile file for the Oracle database
instance and is specified in the Pfile attribute.
• Offline: Uses the shutdown command to stop the database
• Monitor: Scans the process table for ora_dbw, ora_smon, ora_pmon, and
ora_lgwr
The ora_pmon process monitors all other Oracle processes.
If detail monitoring is enabled, as shown in the “Configuring Oracle Detail
Monitoring” section, the monitor script attempts to write to the database.
• Clean: Forcibly stops the Oracle database by using the following svrmgrl or
sqlplus shutdown abort command:
If the process does not respond to the shutdown command, the agent scans
the process table and kills processes associated with the configured instance.
• Info: Provides static and dynamic information about the state of the database
• Action: Performs predefined actions on a resource
Configuring an Oracle Resource
Starts and stops an
Oracle instance
Monitors the process
table
OraSG_OraResource Name
/oraHome
oracleOwner
APSID
Required Attributes
OracleResource Type
OraSGService Group Name
Sample ValueResource Definition
Oracle OraSG_Ora (
Sid = AP
Owner = oracle
Home = "/ora"
)
Oracle OraSG_Ora (
Sid = AP
Owner = oracle
Home = "/ora"
)
main.cfmain.cf
Note: Pfile is a required attribute for versions of the agent prior to 4.1.

Optional Oracle Attributes
The StartUpOpt Attribute
The Oracle agent supports an optional StartUpOpt attribute to enable you to
control how VCS starts a database instance.
Examples:
• RECOVERDB: Assume that in the event the database node fails for a
nondatabase issue (for example, if the system panics due to a hardware issue),
you do not want the database to start up (come online) if any data file recovery
needs to take place. If you set StartUpOpt to RECOVERDB and the system
running the database faults, VCS starts the database in recovery mode when it
fails over to another node.
• CUSTOM: You might want to start the database up in standby mode and apply
logs. In this case, you can create an SQL script to perform these actions, and
this script is called when you set StartUpOpt to CUSTOM.
You must create the script in /opt/VRTSvcs/bin/oracle with the name
of start_custom_SID.sql, where SID is the same as the value of the SID
attribute.
Optional Oracle Attributes: The StartUpOpt
Attribute
You can use the StartUpOpt attribute to specify
how the Oracle instance is started.
ActionValue
Runs the SQL script in a specified location; an
example might be to start up in standby mode and
apply logs
CUSTOM
Performs a recovery of database files upon instance
startup
RECOVERDB
Starts the database in restricted modeRESTRICTED
If the Pfile attribute is empty, run: startup
If Pfile is specified, run:
startup pfile=‘pfile_location’
STARTUP
(default)

2
The ShutDownOpt Attribute
You can configure the ShutDownOpt attribute for the Oracle resource to control
how VCS stops a database instance when the Oracle resource is taken offline.
Note: If CUSTOM is specified, a script must exist in /opt/VRTSvcs/bin/
oracle with the name of shut_custom_SID.sql, where SID is the same as
the value of the SID attribute for this Oracle resource.
The ShutDownOpt Attribute
You can use the ShutdownOpt attribute to specify
how the Oracle instance is stopped.
Uses srvctl to stop the databaseSRVCTLSTOP
Runs the SQL script in a specified locationCUSTOM
Performs a transactional stopTRANSACTIONAL
Runs shutdown immediateIMMEDIATE
(default)
ActionAttribute

Other Optional Oracle Attributes
The Oracle agent supports three other optional attributes described in the slide.
Example main.cf entries showing sample values for these attributes are shown
in the following resource definition. For more information, see the Enterprise
Agent for Oracle User’s Guide for your platform.
Oracle OraSG_Ora (
Sid = AP
Owner = oracle
Home = “/oracle”
Pfile = "/oracle/dbs/initVRT.ora"
EnvFile = "/oracle/.ora_envfile"
AutoEndBkup = 0
Encoding = eucJP
)
The example value for the Encoding attribute sets encoding to the Japanese
language set.
Other Optional Oracle Attributes
AutoEndBkup: Takes the data files in the
database out of the backup mode when
coming online (The default is 1, true.)
EnvFile: Specifies the location of a file
sourced by Oracle agent entry point scripts
containing environment variables
Encoding: Specifies encoding for other
language sets

2
Configuring Oracle Detail Monitoring
The example configuration shows an Oracle resource with detail monitoring
configured.
These optional Oracle attributes are used to configure detail monitoring:
• DetailMonitor: A flag to enable and disable detail monitoring (VCS 4.x)
The default is 0, do not perform detail monitoring. You can set this value
higher than zero to control how often detail monitoring is performed. For
example, if set to 5, detail monitoring is performed every fifth monitor interval.
• User: A database user account with update privileges
• Pword: The password for the database user account
You must encrypt the password as described in a later section.
• Table: The name of the database table VCS uses for additional monitoring
• MonScript: The executable script file containing the SQL statements VCS uses
when writing to the table
• EnvFile: The file containing environment variables sourced by the agent
Configuration Prerequisites
• Create the database user and password for use by VCS.
• Create a test table within the monitored database.
• Create an executable script with SQL statements.
In this example, the user scott with the password tiger should be defined in
the VRT database with update privileges to the table called testtable. This table
should be created in the database before the additional monitoring is enabled.
Configuring Oracle Detail Monitoring
Oracle OraSG_Ora (
Sid = AP
Owner = oracle
Home = “/ora”
Pfile = “/ora/dbs/initAP.ora”
EnvFile = “/ora/.ora_envfile”
DetailMonitor = 1
User = “scott”
Pword = “lgLimHmmKum”
Table = “testtable”
MonScript = “./bin/Oracle/SqlTest.pl”
)
User defined in AP
database with update
privileges for testtable
User defined in AP
database with update
privileges for testtable
The table created in
AP database before
enabling the VCS resource
The table created in
AP database before
enabling the VCS resource
The executable script
containing SQL write
statements
The executable script
containing SQL write
statements
main.cfmain.cf
EnableEnable

Encrypting Passwords
You can use the VCS encryption utility to encrypt database passwords before
configuring the Pword attribute in the Oracle agent configuration.
Note: The value of Pword is automatically encrypted when you use the Oracle
configuration wizard or one of the VCS GUIs to configure the resource.
To encrypt a password, run the vcsencrypt command. For example, type:
# root:/opt/VRTSvcs/bin> vcsencrypt -agent

2
Error Handling for Detail Monitoring
You can configure how VCS responds to different types of errors when detail
monitoring is configured to prevent the database from failing over in cases where
that action is not appropriate.
For example, if the user name and password specified for an Oracle resource is
incorrect, an error of the form ORA-01017 invalid username/password;
logon denied occurs. This error indicates that the configuration is wrong, but
the database may be functioning properly. In this case, you do not want VCS to
bring down the database and restart it on another system. The same failure occurs
wherever the database is brought online.
Rather, you want VCS to notify the appropriate administrator to fix the
configuration problem.
You can customize error handling by modifying the oraerror.dat file in
/opt/VRTSvcs/bin/Oracle. Entries in this file have the format:
Oracle_error_string:action:confidence_level
This example entry leaves the Oracle resource online and sets the confidence level
to 10 when an invalid user name error code is returned.
01017:WARN:10
Error Handling for Detail Monitoring
You can configure the optional Oracle error handling file when
using detail monitoring to specify VCS response based on the
type of database errors. See
/opt/VRTSvcs/bin/Oracle/oraerror.dat.
Marks the resource state as ONLINE and notifies
the administrator (when notification is configured)
WARN
Marks the resource offline when a serious error
occurs; resource faults and SG fails over
FAILOVER
Freezes the service group temporarily and marks
the resource OFFLINE
NOFAILOVER
Marks the resource state as UNKNOWNUNKNOWN
Error is not critical; take no actionIGNORE
DescriptionAction

Configuring the Netlsnr Resource and Agent
The following main.cf excerpt corresponds to the Netlsnr example resource
shown in the design worksheet in the slide.
Netlsnr OraSG_Listen (
Owner = oracle
Home = "/oracle"
TnsAdmin = "/ora/net/admin"
Listener= LISTENER
LsnrPwd = S2cEjcD5s3Cbc
EnvFile = "/ora/dbs/envfile“
MonScript = "./bin/Netlsnr/LsnrTest.pl"
)
Note: The Sqlnet resource type manages the listener for versions of the Oracle
agent prior to 4.0. The attributes are the same, except LsnrPwd, which is not
supported prior to 4.0.
/oracle/dbs/envfileEnvFile
LISTENERListener
S2cEjcD5s3CbcLsnrPwd
Optional Attributes
OraSG_ListenResource Name
./bin/Netlsnr/Lsnr
Test.pl
Monscript
/ora/net/adminTNSAdmin
/oraHome
oracleOwner
Required Attributes
NetlsnrResource Type
OraSGService Group Name
Configuring a Netlsnr Resource
Starts and stops a
listener process
using lsnrctl
Monitors the process
table
Detail Monitoring

2
Agent Entry Points
• Online: Uses the following command to start the listener process:
lsnrctl -start $LISTENER
• Offline: Uses the following command to stop the listener process:
lsnrctl -stop $LISTENER
• Monitor: Scans the process table for tnslsnr $LISTENER and runs
MonScript as the option
If secondary monitoring is enabled, the monitor script runs the command:
lsnrctl -stat $LISTENER
• Clean: Scans the process table for tnslsnr $LISTENER and kills it
• Action: Performs predefined actions on a resource

Common Oracle Configurations
The diagram shows some common Oracle VCS configurations.
• In the resource diagram on the left:
– Oracle data files are written to raw volumes. A Mount resource is not
required.
– Oracle binary files are located on shared storage on a file system with a
separate volume that is part of the service group configuration, so they are
highly available.
– The Oracle instance must be online before the listener is started.
• In the resource diagram on the right:
– The Oracle resource must come online last.
– The log and data files are located on separate file systems and volumes, but
they use the same disk group.
– The binaries are on local storage and so are not part of the service group
configuration.
Note: As a best practice, minimize the number of volumes and disk groups used in
database service groups. As a general rule, use one disk group per database. Large
numbers of volume management objects complicate administration and can impact
performance during startup.
Common Oracle Configurations
Oracle
Binary
Volume
Mount
Netlsnr
NIC
IP
Disk
Group
Data
Volume
Oracle
Disk
Group
Log
Volume
Mount
Netlsnr
NIC
IP
Data
Volume
Mount

2
Alternative Oracle Configurations
You can use variations of resources and dependencies in an Oracle service group to
accommodate different configurations. Consider these alternatives:
• The Oracle resource is not required to be dependent on the IP resource.
However, there is little purpose in bringing the database online unless the IP
address is available and clients can connect to the database.
• Regardless of whether you link the Oracle resource to the IP resources, the
Netlsnr resource should have a dependency on the IP resource.
• If Net8 is used, a Netlsnr resource can be configured with no dependency on
the database. The Net8 listener can be started independently of the database.

Database Failover Behavior
Database Failover Scenarios
You can use resource and service group attributes to control how VCS responds
when resources or systems fault. A complete discussion of failover behavior is
provided in the “VCS Response to Resource Faults” and “Handling System and
Communication Failures” lessons in the VERITAS Cluster Server Fundamentals
course.
The scenarios in the slide show the practical application of some of the more
common failover configurations.
Database Failover Scenarios
If you want the listener to be restarted and never
cause failover if it faults:
– Set the listener resource to non-critical.
– Do not make the listener a child of a critical resource.
– Set RestartLimit = x.
If you do not want VCS to forcibly clean or fail over if
the database or listener faults:
– Set ManageFaults=None for the service group.
– Notification on fault, no clean, no failover
Even if a database resource is non-critical, the
clean entry point is run if the resource faults.
This shuts down the database and leaves it offline.
Even if a database resource is non-critical, the
clean entry point is run if the resource faults.
This shuts down the database and leaves it offline.
!

2
Failover During Oracle Hot Backup
If a cluster node fails while running a hot backup of an Oracle database, failover to
another node can succeed only if the Oracle resource attribute, AutoEndBkUp, is
set to a nonzero value.
Otherwise, the database in the backup mode on the failover system cannot be
opened and VCS cannot bring the Oracle resource online.
The following errors are displayed to indicate this condition:
$ ORA-1110 "data file %s: ’%s’"
or
$ ORA-1113 "file %s needs media recovery"
Before VCS can bring the Oracle resource online on the failover system, you must
take the data files in the database out of the backup mode and shut down the
database so that it can be reopened. Refer to the Oracle documentation for
instructions on how to change the state of the database files.
Failover During Oracle Hot Backup
If a fault occurs while a database is in hot
backup mode, you can prevent the service
group from coming online elsewhere.
Set the AutoEndBkUp attribute to 0 to prevent
the database from being opened on another
node.
By default, AutoEndBkUp is set to 1, which
takes data files out of backup mode when the
Oracle resource is brought online, enabling
failover.

Additional Oracle Agent Functions
In 4.x versions, the Oracle agent supports two additional entry points you can use
to manage database functions from within VCS:
• Info: Checks the status of the instance
• Action: Performs specified actions, such as backing up the Oracle database,
changing the database state, and suspending and resuming a database
You can initiate these types of operations using the hares command, as shown on
the following pages.
This can be useful for scripting common database administration tasks that can be
initiated from the VCS operator or administrator.
Other Oracle Agent Functions
You can use the info and action entry points of
the Oracle agent to perform additional tasks.
Info: Provides static and dynamic information about
the state of an online resource (such as a database)
Action: Enables you to perform predefined actions on
a resource
These entry points are supported in other resource
types as well.
Their uses related to Oracle are described in more
detail in the following sections.

2
Predefined Actions for Oracle
The Oracle agent supports the action entry point, which enables you to perform
predefined actions on a resource.
The Oracle agent supports the actions shown in the slide. The Netlsnr agent
supports two of these actions:
• VRTS_GetInstanceName
• VRTS_GetRunningServices
Predefined Actions for Oracle
Resumes a suspended databaseDBResume
Suspends a database (Oracle 8i and later)DBSuspend
Enables RESTRICTED mode for the database sessionDBRestrict
Disables RESTRICTED mode for the database sessionDBUndoRestrict
Places a tablespace in backup modeDBTbspBackup
Retrieves the list of processes monitored by the agentVRTS_GetRunning
Services*
Retrieves the name of the configured instanceVRTS_GetInstance
Name*
DescriptionKeyword (Action)
*Also applies to Netlsnr resources

Initiating Actions
To perform an action on a resource, use the hares command with the following
syntax:
hares -action res token [-actionargs <arg1 ...>] [-sys
system]
The keyword represented as token in the command line example, must be one of
the keywords shown in the left column on the “Predefined Actions for Oracle“
slide. Example command lines are shown in the slide.
Custom actions can be developed enabling you to perform additional actions. See
the VERITAS Cluster Server Agent Development Guide for more information.
Initiating Actions
Use the hares command to initiate predefined
actions on a resource.
Example usage:
hares -action OraSG_Ora DBRestrict
hares –action OraSG_Ora DBTbspBackup
-actionargs mytblspace
You can also develop custom actions for an
agent.

2
Retrievable Information Using the Info Entry Point
The Oracle agent info entry point provides static and dynamic information about
the state of the database. As described further in the next section, you can use the
hares command to obtain predefined types of information shown in the tables on
the slide.
When invoked, the info entry point maps one of these keywords to a SQL
statement and runs that statement on the database specified by the Oracle resource.
Retrievable Information Using the Info Entry
Point
InstanceRoleThreadParallel
StartupTimeHostNameDatabaseName
InstanceNameInstanceNoVersion
Shared Pool
Percent free
FreeDBfileSizeTotalDBfileSize
Buffer Hits Percent
DatabaseStatusShutdownPendingLogMode
OpenModeLoginsInstanceStatus
Static Information
Dynamic Information

Retrieving Information
You can manually invoke the info entry point using the hares command.
• The first command line shown in the slide causes the agent to update the value
of the ResourceInfo attribute for the specified resource on the specified system
if the resource is online. If the Info entry point is successful, no output is
displayed.
Note: ResourceInfo is a temporary attribute—its values are never dumped to
the main.cf file.
• The second command line displays the value of the ResourceInfo attribute.
As an alternative to manually running hares -refreshinfo, you can set the
InfoInterval resource type attribute to a nonzero value. This tells the agent to run
the info entry point periodically, according to the value of the InfoInterval
attribute.
To set the InfoInterval type attribute, use the following syntax:
hatype -modify Oracle InfoInterval seconds
Note: You can add SQL statements to the /opt/VRTSvcs/bin/Oracle/
resinfo.sql file to obtain additional information using the info entry point. For
information on how to customize this file, see the VERITAS Cluster Server
Enterprise Agent for Oracle, Installation and Configuration Guide.
Retrieving Information
Use the hares command to retrieve information
about an online resource.
Example usage:
hares –refreshinfo OraSG_Ora –sys sys_name
hares –value OraSG_Ora ResourceInfo
You can also add SQL statements to the
following file to customize information retrieval
using the info entry point:
/opt/VRTSvcs/bin/Oracle/resinfo.sql

2
Summary
This lesson described how to put a database service under control of VCS. An
Oracle example was presented to show the resource types and agent functions that
are provided with enterprise agents.
Next Steps
Now that you understand how Oracle agents manage exported file systems, you
can configure other common types of high availability services.
Additional Resources
• VERITAS Cluster Server Bundled Agents Reference Guide
This guide describes each bundled agent in detail.
• VERITAS Cluster Server Enterprise Agent for Oracle, Installation and User’s
Guide
This guide provides detailed information on procedures and concepts for
configuring and managing Oracle using the VCS enterprise agent.
• VERITAS Cluster Server Agent Developer’s Guide
This guide describes how to customize agents, including the info and action
entry points.
• High Availability Design Using VERITAS Cluster Server Course
This provides guidance on designing local and global clusters to manage
database environments.
Lesson Summary
Key Points
– VCS provides a variety of agents to manage
databases.
– The Oracle agent has specialized functions for
managing Oracle databases.
Reference Materials
– VERITAS Cluster Server Bundled Agents
Reference Guide
– VERITAS Cluster Server Enterprise Agent for
Oracle, Installation and Configuration Guide
– VERITAS Cluster Server Agent Developer's Guide
– High Availability Design Using VERITAS Cluster
Server Course

Lab 2: Creating Oracle Resources
Labs and solutions for this lesson are located on the following pages.
Appendix A provides brief lab instructions for experienced students.
• “Lab 2 Synopsis: Creating Oracle Resources,” page A-6
Appendix B provides step-by-step lab instructions.
• “Lab 2 Details: Creating Oracle Resources,” page B-11
Appendix C provides complete lab instructions and solutions.
• “Lab 2 Solutions: Creating Oracle Resources,” page C-11
Goal
The purpose of this lab is to make an Oracle database service highly available.
Results
Each student’s database service can be started, monitored, and stopped on each
cluster system.
Prerequisites
Obtain any classroom-specific values needed for your classroom lab environment
and record these values in your design worksheet included with the lab exercise
instructions.
Lab 2: Creating Oracle Resources
name
Oracle1
name
Mount1
nameVol1
nameDG1
nameProxy1
nameIP1
name
App1*
nameSG1nameSG1
1. Add an Oracle
resource.
2. Add a Netlsnr
resource.
1. Add an Oracle
resource.
2. Add a Netlsnr
resource.
name
Listener1
See the next slide for classroom values.See the next slide for classroom values.
*This may be a
Process resource if
the Application lab
was not completed.

Introduction
Overview
This module describes a recommended practice to prepare resources and manually
test NFS services before configuring an NFS service group. In this lesson, you are
introduced to the VCS Share and NFS resource types.
Importance
Although understanding how agents work enables you to use any VCS resource
type, you need to have an understanding of the specific application service at hand
to successfully configure the application in a high availability environment.
Understanding how you can use VCS to manage your NFS shares helps you
implement highly available NFS servers.
Outline of Topics
• NFS Service Overview
• Preparing NFS for High Availability
• Testing the NFS Service
• Configuring an NFS Service Group
• NFS Lock Failover in VCS 4.1
Configuring NFS and Share resources
in an NFS service group.
Configuring an NFS
Service Group
Configure an NFSLock resource to fail
over record locks.
NFS Lock Failover in VCS
4.1
Test the NFS service on each target
system.
Testing an NFS Service
Prepare NFS components for the VCS
environment.
Preparing NFS for High
Availability
Describe how NFS services work in a
high availability environment.
NFS Service Overview
will be able to:
Topic

Example 3 Clustering NFS 3–3
3
Network File System (NFS) provides file system services to remote clients over
the public network. As with most application services, NFS service components
can be categorized into three basic components:
• Application—the NFS processes (daemons) that handle file system mount
requests (mountd), client requests (nfsd), and the share name configuration
The diagram shows a simple example of an NFS service.
NFS daemons
Share
Client Network
Requests
IP Address
NIC
File System
Volume
Disk Group
NetworkNetwork
StorageStorage

NFS Configuration and Migration Process Overview
Use the process shown in the diagram to prepare and test NFS services on each
system before putting them under VCS control. Use the design worksheet to obtain
and record information about the service group and each resource. This is the
information you need to configure VCS to control these resources.
Details are provided in the following sections.
NFS Configuration and Migration Process
Ready for
VCS
Ready for
VCS
More
Systems?
More
Systems?
N
Y
Perform one-time
configuration tasks on
each system.
Perform one-time
configuration tasks on
each system.
Start, verify, and
stop NFS services on
one system at a time.
Start, verify, and
stop NFS services on
one system at a time.
Create or prepare NFS-related resources initially on each
system for each NFS service that will be put under VCS control;
this is not part of VCS operation.
Manually test NFS shared file systems on each system that is a
startup or failover target for that service group before putting
the file systems under VCS control.

3
Preparing NFS for High Availability
Identifying NFS Components and Relationships
VCS manages NFS services in the same way it manages any other application
service. Resources are contained within the service group for components, and
dependencies are defined to ensure that VCS brings the supporting resources
online in the necessary order.
The diagram shows the resources in an NFS service and their relationships. You
can use this diagram as a guide to configure and test NFS resources before
bringing them under VCS control.
Notice that unlike many other resource dependency diagrams, the IP resource is
the topmost parent. If the IP address came up before the NFS shared file system is
available, a client attempting to connect to the share gets a stale file handle and
never tries to mount the file system again. However, if the IP address is not
configured yet, the client retries to connect to the file system.
Identifying NFS Components and Relationships
DiskGroup
NIC
IP
Mount
Share
NFS
Volume
Shared storage
resources:
– Disks or components
of a logical volume
manager
– File systems to be
mounted
Network resources:
– IP addresses
– Network interfaces
Application (NFS)
resources:
– NFS (nfsd, mountd)
– Share names

Configuring Shared Storage for NFS
After you create the disk group, volume, and file system to be shared, verify that
the major and minor devices number for storage devices are the same on each
system.
NFS services use the major and minor number of the block device to identify the
exported file system. For the identity of the exported file system to be the same no
matter which system in the cluster provides NFS services, the major and minor
numbers of the storage device must be the same on each system in the cluster.
Generally, major and minor numbers match. However, the values are affected by
circumstances, such as the order in which software is installed. For example, if you
install VxVM and VxFS in different orders on different systems, the major
numbers for devices can be different.
Configuring Shared Storage for NFS
Major and minor numbers for block devices used
for NFS services must be the same on each system.
Stale File Handle
Error
NFS
Request
NFS
Request
NFS
Response
Before failover After failover

3
Major Numbers for Volumes
To display and change the major numbers on the VERITAS Volume Manager
(VxVM) volumes on shared storage, use the following platform-specific
instructions.
Solaris
1 Find the number corresponding to vxio and vxspec in the
/etc/name_to_major file. Type:
grep vx /etc/name_to_major
2 Select major numbers that are not in use by any other device by viewing the
/etc/name_to_major file on each system.
3 To make the major numbers the same on both systems use the haremajor
command to change the major numbers for vxio and vxspec on S1 to match
S2. For example, type:
haremajor -vx 90 91
Note: It may be necessary to change the major numbers on more than one
system to make them unique for all systems.
4 Reboot each system on which haremajor was successful changing major
numbers.
Repeat this procedure for each volume used for an NFS file system.
Major Numbers for Volumes
Verify that each system has the same major number
for the block device of the shared volume.
Verify that major numbers are also unique within a
system.
Verify that the major numbers match on all systems.
If major numbers do not match, reset one system’s
numbers to match the other system.
vxdmp 87
vxio 88
vxspec 89
vxdmp 87
vxio 88
vxspec 89 S1
vxdmp 89
vxio 90
vxspec 91
vxdmp 89
vxio 90
vxspec 91 S2

AIX
Use the lvlstmajor command to find out what the available major numbers are
for the file system. On each system:
1 Check the vxio driver numbers. Major numbers match the driver number.
# haremajor -v
47
2 Find an unused driver number on all systems.
# haremajor –a
50...
3 Change each system to the same driver number.
# haremajor -s 50
vxio major number changed to 50
4 Reboot each system.
# reboot
HP-UX
See the HP-UX system documentation for information about changing LVM
device numbers.
Linux
Major and minor device numbers on Linux systems depend entirely on the order of
the SCSI devices—internal and external are found during system startup.
Therefore, you cannot ensure that major and minor numbers match on different
Linux systems.
Because of this platform-specific limitation, your clients may have problems with
stale file handles to NFS-mounted file systems after a failover. To mitigate this
issue, follow these guidelines:
• Ensure that shared storage is on a different controller (than the internal
controller) on each node.
• Verify that the SCSI driver for the mentioned controller is loaded after the
internal controller (if such a driver exists).
• If both controllers are driven by the same driver then ensure that the internal
controller is on a lower PCI slot.

3
Minor Numbers for Volumes
Use the ls command to determine whether the minor numbers for volumes match
on each system.
ls –l /dev/vx/rdsk/dgname/volume_name
For example:
ls –l /dev/vx/rdsk/datadg/vol01
crw------- 1 root root 99 0x000005 Aug 17 07:30 /dev/vx/
rdsk/datadg/vol01
If the minor numbers of the volume devices do not match, change the base minor
number for the disk group to a common value on all systems:
vxdg reminor dgname new_base_number
For detailed information on using haremajor and vxdg reminor, refer to the
VERITAS Cluster Server Installation Guide. You can also refer to the Volume
Manager command vxdg manual page for instructions on reconciling the
VERITAS Volume Manager minor numbers.

Configuring Networking for NFS
In a high availability environment, the IP address and host name that are used by
the client to access the NFS service should not be tied to a specific system because
the same service can be provided by any system in the cluster.
Ensure that you have an administrative IP address configured on the network
interface that clients will use to access the NFS shared file systems. The
administrative IP address must be configured to enable VCS to set up the virtual IP
address for the NFS service group.
Set up any other network configuration files needed for your public network
environment. Also, configure the name service to enable clients to connect to the
shared file system using a virtual host name and IP address that will migrate
among the systems that can host the NFS service.
Configuring Networking for NFS
An administrative IP address must be
configured during system startup for the
interface used by this service.
Add entries to the name server to enable
clients to access the virtual IP address and
host name used to connect to the shared file
system.
Set up any other applicable network
configuration files.

3
Configuring NFS (Application) Components
Checking the Startup Configuration
To share a file system in a high availability environment where the file system can
be mounted and shared from any one of the systems, you must prevent the
operating system from controlling the NFS resources:
Solaris
• The file system mount point should not have an entry in the /etc/dfs/
dfstab file.
• The file system should not have an entry in the /etc/vfstab file or the entry
should say no to mount at boot.
AIX
• The file system mount point should not have an entry in /etc/exports.
• The file system should have an entry in /etc/filesystems with
mount=false.
HP-UX
• The file system should not have an entry in the /etc/fstab file.
Linux
• The file system should have an entry with noauto in /etc/fstab.
Checking the Startup Configuration
Verify that the file system is not mounted or shared
at system startup to enable VCS control:
Solaris:
– No file system mount point entry is placed in
/etc/dfs/dfstab.
– The /etc/vfstab entry should say no or manual
mount at boot.
AIX: No entry in /etc/exports; a mount=false entry
/etc/filesystems
HP-UX: No entries in /etc/exports or /etc/fstab
Linux: No entries in /etc/exports; noauto entry in
/etc/fstab

Testing the NFS Service
Before configuring a service group in VCS to manage an NFS service, test it on
each system that can be a startup or failover target for the service group. Following
this best practice recommendation ensures that VCS will successfully manage the
shared file systems when you configure the service group.
Detailed information about setting up shared storage and network resources was
provided earlier in the course. This lesson focuses on the additional steps needed
for testing NFS services.
Testing Procedure
Bring up resources.Bring up resources.
Ready for
VCS
Ready for
VCS
More
Systems?
More
Systems?
Bring up resources.Bring up resources.
N
Y
Test remote access.Test remote access.
S1
S2…Sn
Test remote access.Test remote access.
Shared storageShared storage
Start up all resources
in dependency order.
NFS daemons, exportsNFS daemons, exports
Virtual IP addressVirtual IP address

3
Starting NFS Daemons
The NFS daemons are usually configured to start during system boot. To verify the
appropriate daemons are running, and start them, if necessary:
Solaris
ps -ef | grep nfs mountd statd lockd
/usr/lib/nfs/nfsd -a nserver
/usr/lib/nfs/mountd
/usr/lib/nfs/statd
/usr/lib/nfs/lockd
AIX
lssrc –g nfs
startsrc –g nfs
HP-UX
ps -ef | grep nfs
/usr/sbin/nfsd -a nserver
/usr/sbin/rpc.mountd
Linux
/sbin/service nfs status
/sbin/service nfs start
/usr/sbin/rpc.mountd
Starting NFS Daemons
Verify that the NFS daemons are running or start
them if necessary:
1. Determine whether the appropriate NFS daemons
are running:
Solaris: nfsd, mountd, statd, lockd
AIX: nfsd, rpc.mountd, rpc.lockd, rpc.statd
HP-UX: nfsd, rpc.mountd, rpc.lockd, rpc.statd
Linux: nfsd, mountd
2. Start the NFS daemons, if they are not already
running.

Exporting the File System
After you have verified that the file system is mounted and the NFS daemons are
running, export (share) the mount point of the file system with specified share
options, if any. Then, verify that the share is configured.
Solaris
share mount_point
share | grep mount_point
AIX
smitty mknfsexp: creates/updates /etc/exports
exportfs –a
showmount
HP-UX
/usr/sbin/exportfs [–F fstype] mount_point
exportfs | grep mount_point
Linux
/usr/sbin/exportfs [–F fstype] mount_point
exportfs –a
Note: Whenever you change /etc/exports, you must notify the NFS daemon
using either of these methods:
• /sbin/service nfs reload
• /sbin/exportfs -r
Exporting and Testing the File System
After verifying that the file system is mounted and
NFS daemons are running:
1. Export the file system.
2. Verify that the file system is exported.
3. Configure the virtual IP address.
4. Mount the file system from a remote client on
the public network.

3
Migrating a Share
Before you place an NFS service under VCS control, it is recommended that you
manually migrate the service between all intended target systems. By doing so,
you:
• Ensure that your operating system resources are properly configured on all
potential target cluster systems.
• Observe the types of tasks VCS performs to bring a service online, monitor the
status of each resource, and switch a service to another system.
When you have completed the preparation of the testing tasks, you can document
or verify that your design worksheet is complete, and then configure VCS to
manage the NFS services.
Migrating a Share
NetworkNetwork
StorageStorage
S1
S2
NFS daemons
ShareFile System
Volume
Disk Group
IP Address
NIC

Configuring an NFS Service Group
Use the online or offline configuration process described earlier in the course to
create a service group and add resources after you have manually tested the NFS
configuration on each system.
The NFS Resource
Entry Points
• Online: Checks whether the nfsd and mountd processes are running
If processes are not running, the online entry point starts them and exits.
• Offline: None
• Monitor:
– Solaris, HP-UX, Linux: Monitors TCP and UDP versions of the processes
by sending RPC calls clnt_create and clnt_call to the RPC server
If calls succeed, the resource is reported as online.
– AIX: Determines whether processes are active by querying the SRC
Optional Attributes
• Nservers: Specifies the number of concurrent NFS requests the server can
handle
Defaults: Solaris 16; HP-UX 4; AIX 0, obtains the value from the SRC
• Nproc (Linux-only): Specifies the number of processes of nfsd to start
The default value is 8.
• Protocol (HP-UX only): Specifies the UDP or TCP protocol
The NFS Resource
The NFS agent:
Starts the NFS processes if they are not already running
No offline; OnOnly resource
Checks for active NFS-related processes (based on platform
type)
APExportNFSResource Name
32Nservers
Optional Attributes
NFSResource Type
APExportSGService Group Name
NFS APExportNFS (
Nservers = 32
)
NFS APExportNFS (
Nservers = 32
)
main.cfmain.cf

3
Configuring a Share Resource
Entry Points
• Online: Shares an NFS file system
• Offline: Unshares an NFS file system
• Monitor: If there is an entry in the shared table for the file system specified by
PathName, the resource is reported as online.
Optional Attributes
Options: Specifies options to be used with the share command while sharing the
file system. Defaults are:
Solaris, AIX, HP-UX
No options are specified by default.
Linux
ro, async, wdelay, root_squash
Note: Linux has an additional required attribute, Client, which specifies the host to
which the PathName directory is exported.
Configuration Prerequisites
• The file system and share should not be configured automatically during
system startup.
• The Mount and NFS resources must be configured in VCS.
Configuring a Share Resource
The Share agent:
Shares and unshares an NFS file system
Reads the share table to check for an entry for the file system
/AP/export1PathName
Required Attributes
APExportShareResource Name
Options
Optional Attributes
ShareResource Type
APExportSGService Group Name
Share APExportShare (
PathName = "/AP/export1"
)
Share APExportShare (
PathName = "/AP/export1"
)
main.cfmain.cf

NFS Considerations in a VCS Environment
NFS-specific issues to consider when configuring a high availability NFS service
group include:
• Increase RestartLimit to avoid failover when daemons fail sporadically.
• Increase the Nservers value if the NFS daemon needs more threads. If you
need to increase this value after the service group is running:
a Freeze the service group.
b Change the Nservers attribute value.
c Stop and restart NFS daemons outside of VCS to restart with the increased
Nservers value.
Note: You cannot simply take the NFS resource offline and bring it back
online. The NFS daemons are not stopped when you take the NFS resource
offline because the NFS resource type is on-only.
d Probe the service group to clear the faulted NFS resource.
e Unfreeze the service group.
• Avoid using a cluster system that can be a failover target for the NFS service as
a client to mount the shared file system.
If you have the file system mounted and the NFS service group fails over to
that system, the client is stopped as a prerequisite to starting the NFS server.
Also, it is generally good practice to avoid relying on services outside the
cluster.
NFS Considerations in a VCS Environment
Scenarios:
Consider setting RestartLimit to a higher value than
the default of 1 so that VCS attempts to restart the
NFS daemons before failing over to another system.
Increase the value of Nservers (Nproc on Linux) if
you need to increase the number of client
connections.
Note: If you increase this attribute while the resource
is online, freeze the service group, stop and restart
NFS outside of VCS, probe, and then unfreeze the
service group.
Best practice recommends that you do not use
cluster nodes as NFS clients.

havcs-410-101 a-2-10-srt-pg_3

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à havcs-410-101 a-2-10-srt-pg_3

Similaire à havcs-410-101 a-2-10-srt-pg_3 (20)

Dernier

Dernier (20)

havcs-410-101 a-2-10-srt-pg_3