Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Tips on successfully upgrading to apps 11i for the non apps dba
1. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
Introduction
1The methods in this paper are based on the experiences of upgrading Oracle
Applications 10.7 NCA (database version 7.3.4) to Oracle Applications 11.5.3 (database
version 8.1.6). The target system is a Windows NT dual 500 MHz CPU server with one
Gigabyte of RAM and 54Gb of available disk space. The Applications installation
upgraded had nearly no customization. Some of the suggestions presented below can be
applied to other types of database upgrades and to other platforms.
Background
This paper is presented by Christina Zanoni, the Database Administrator at Attorneys’
Liability Assurance Society, Inc., A Risk Retention Group (ALAS). ALAS is a mutual
insurance company that offers the highest quality professional liability insurance to a
select group of large law firms. ALAS insures over 52,000 lawyers from the most
prestigious law firms in the U.S. and is rated “AA” by both Standard and Poor’s and
Fitch rating agencies. Our small size (fewer than 100 employees worldwide) makes it
effective for us to manage our computing environment with in-house staff. In addition,
we are able to manage our Oracle Applications installation internally because of the few
customizations we have made to it. Since we have just one DBA, she gets involved with
Oracle Applications activities that a DBA at a larger firm with several DBAs may not be
called upon to perform.
Oracle Applications is a set of financial management and accounting applications, called
modules or products, which are integrated within a single database. Oracle Applications
primarily uses stored procedures and functions to perform applications tasks along with
Forms and Reports. Additionally, supporting code exists in the form of executable files.
Oracle Applications uses a multi-tiered approach to organizing this structure, which is
explained below. Generally, one can refer to the database, stored procedures and
executables as the “database tier”; the Forms and Reports as the “middle tier”; and the
web browser as the “desktop tier.” Oracle Applications also relies upon a set of external
“concurrent managers” that are execution queues to process reports and complex tasks.
ALAS uses only those Oracle Application modules required to manage the general
ledger, record accounts receivable and accounts payable data, and track fixed assets. In
all, Oracle Applications provides over 150 products, and many sites make use of far more
products than does ALAS. As mentioned above, the ALAS installation of Oracle
Applications involves almost no customization. Typically, a site will customize a number
of reports, forms, and even code objects to provide appropriate functionality for their
environment. Due to this, most Applications upgrades require a great deal of time and
effort converting and re-customizing this functionality. Given this, it is appropriate to
think of this paper as presenting a fairly “generic” upgrade to Oracle Applications.
Upgrade Preparation/Implementation Suggestions
Christina M. Zanoni Page 1 4/5/2012
2. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
Overview/Understanding the Process
Oracle Applications modules involve data and code stored in individual schemas that are
dependent on other more general schemas such as the Applications Object Library
(AOL). The AOL schema stores code objects that are used by many different modules.
The AOL schema forms the primary core of an Applications database and is owned by
the Applications System account. Oracle Applications has two primary system accounts:
APPLSYS and APPS, these are analogous to the SYSTEM and SYS users. The AOL
objects are stored in the APPS schema just as Oracle RDBMS objects are stored in the
SYS schema. During an upgrade, the APPS objects, as well as objects in other schemas,
are modified to provide the new functionality, and the Report, Forms, and other
supporting code are also updated. This is similar to Oracle RDBMS upgrade. The major
difference is that, since Applications incorporates code and data storage, the tables and
data they contain must also be upgraded. This is not present during RDBMS upgrades.
The upgrade process (and the patching process described below) is divided into
categories. Each category represents a progression towards the upgraded system, and
indicates the state in which the Applications and RDBMS are left. Each category has a
sequence of steps that either run scripts or involve user interaction. Many steps are not
required for every installation. The category upgrade steps required for
ALAS’Applications 11i upgrade are discussed in some detail later in this paper.
1Oracle Applications 11.5.3 uses “server partitioning” to allow the database to run on an
Oracle 8.1.6 engine and the applications utilities to use Oracle 8.0.6 executables. This
mechanism allows the Applications code to be linked against older utilities, thereby
providing Oracle a way to upgrade portions of the Applications at different times. This is
unlike custom Oracle database applications that would use the same Oracle database
version for the RDBMS and utilities (e.g. SQL*Plus, Oracle Forms).
Server Configuration for Applications 11i
Oracle Applications Release 11i utilizes a three tier architecture consisting of the
database tier, the middle tier and the desktop tier. The underlying technology stack for
Applications 11.5.3 includes Oracle 8i Enterprise Edition (8.1.6.1), Oracle Forms 6i
Server, Oracle Reports 6i Server, Oracle Graphics 6i, Oracle Apache web server, JRE
(Java Runtime Environment), JDK used in context of the web server, and Jinitiator for
the desktop client.
Oracle Applications Release 11i requires Oracle8i on the database tier, and Java is used
on all tiers. The Applications Technology Stack on the Middle Tier is a shared Oracle
8.0.6 Home. Oracle Applications Release 11i requires multiple ORACLE_HOMEs: the
database home (Oracle8i files linked to the Oracle8i database server), the technology
stack home (Oracle8 files linked to the technology stack), and the HTTP home (Oracle8i
files linked to the HTTP server). Split configuration (server partitioning) is used in
Applications Release 11i for multiple Oracle homes. Figure 1 below shows the three tier
architecture, which is followed by a brief explanation of the Applications file structure.
Please read Oracle Applications Concepts Manual for a detailed explanation of the
Christina M. Zanoni Page 2 4/5/2012
3. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
structure of the Applications file system. Much of the information contained in this
section comes from the manual.
FIGURE 1
Database
Tier Database Server (Oracle 8i Enterprise Edition)
Middle Applications Technology Stack (8.0.6 Oracle Home)
Tier
(Includes the Admin, Concurrent Processing, Forms, Reports and Apache Web
Servers.)
Desktop Jinitiator within a standard web
Tier browser.
This section briefly describes each tier and key directories for the data on that tier.
The database server contains the Oracle8i database files used by Oracle Applications
(<dbname>data or <dbname>db directories). Also, the concurrent managers use
temporary files located on the database server.
The application tier server (middle tier) consists of the Oracle Applications product files
(<dbname>APPL), the Java, HTML, log, and output files (<dbname>COMN), and the
Oracle Homes for Apache Server and Oracle technology stack (<dbname>ORA). The
middle tier typically consists of the Admin, Concurrent Processing, Forms, Reports and
Apache Web server.
The <dbname>APPL directory contains the Oracle Applications files and is defined as
the APPL_TOP environment setting. Product directories for all products and the main
environment file (<dbname>.cmd) are in the APPL_TOP directory. The
<dbname>COMN directory contains files that are used by all Oracle Applications
products and is the default location for the log and output directories for concurrent
managers. The <dbname>ORA directory contains the database home, the technology
stack home, and the HTTP home. The database and HTTP homes contain Oracle8i
library and object files. The technology stack home uses files from the Oracle8 server.
No Oracle files are stored on the desktop tier for Oracle Applications Release 11i.
We chose to combine the database and middle tier onto one server due to the small
number of users we support (<10 concurrent users). We used a dual 500 Mhz CPU server
with one Gigabyte of RAM and 54 gigabytes of hard drive space spread over three disk
Christina M. Zanoni Page 3 4/5/2012
4. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
drives. If required, the middle tier components can be moved to one or more other servers
to increase bandwidth. Please read the OAUG white paper written by Milan Rahman
referenced in the Resources section at the end of this paper. This paper contains a
detailed analysis of the advantages of installing the database and middle tier components
onto one server during the upgrade process.
We used three servers to upgrade to Applications 11i: our production database server
(Applications 10.7NCA/Oracle 7.3.4); and interim server with 10.7NCA on Oracle 8.1.6
(Applications 10.7NCA/Oracle8i server); and an empty database server running
Applications 11i (Applications 11i server). We found that using this type of configuration
allowed us to move rapidly from production Applications 10.7NCA/Oracle 7.3.4 to
Applications 11.5.3/Oracle 8.1.6. Figure 2 below lists the servers and a brief description
of their purpose during the upgrade.
FIGURE 2
Applications 10.7NCA/Oracle 7.3.4 database server – This server was our
production database server. It was used to export the full database whenever
required.
Applications 10.7NCA/Oracle8i database server – This server had an Oracle
8.1.6 engine which allowed us to import the database from the previous server
repeatedly without needing to perform the entire database upgrade over.
Category 1-3 steps could be performed on a newly imported database as
required, then the data files were copied to the next server.
Applications 11i server – The server had the 11i system installed “out of the
box” for each iteration. It allowed us to bring over the database from the
previous server to perform the remaining category steps. By not converting
the previous server, time was saved in subsequent iterations. Performing an
installation rather than an upgrade of the previous server, allowed a clean
install of all files.
Flow Charting the Upgrade
Creating a flow chart prior to starting an upgrade will assist you in planning for future
steps and avoiding possible issues. Figure 3 below is a flow chart of the process that we
used to outline the upgrade to Applications 11i.
Christina M. Zanoni Page 4 4/5/2012
5. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
FIGURE 3
Upgrade database to Release
8.1.6.1 using server partition Use Rapid Install to create the
mode new file system on
database/middle tier(s).
Apply all family consolidated
upgrade patches for all products
Perform pre-upgrade steps using
as listed on Oracle MetaLink.
current ORACLE_HOME,
APPL_TOP, and new
APPL_TOPadminpreupg directory
for category 1 and 2.
Perform category 3 steps using current
ORACLE_HOME, APPL_TOP, new
APPL_TOP and new Convert database to Cost Based
APPL_TOPadminpreupg directory. Optimizer and complete remaining final
pre-upgrade steps.
Run AutoUpgrade to upgrade the
Oracle Applications data model and
data. Run the AutoPatch 11.5.3 upgrade
database driver.
Run Rapid Install to configure and start
server processes. Perform post-upgrade category 4 through
6 steps.
Complete post-category steps and apply
any required patches determined through
testing.
The Patching Process of the Upgrade
Applying applications patches during an upgrade can be tedious and lead to additional
issues if you do not have a solid understanding of the patching process. When upgrading
or patching, you are altering operating system files and objects in the database. Oracle
Applications patches usually consist of three patch drivers: a copy driver that copies
operating system files, a database driver that runs SQL statements to modify database
objects, and a generate driver to generate messages, forms or reports. The readme.txt file
of the patch will outline the steps needed to be done in order to successfully apply an
application patch. It is imperative that you follow the steps in detail, including applying
prerequisite patches. The utility used to apply patches is adpatch. Below is a list of
steps to apply patches to an Applications 11i environment on a Windows NT/2000
Server.
1. Back up Oracle Applications and database servers.
2. Shut down all forms services, web listener services, and concurrent manager
services.
3. Verify that the database and database listeners are running.
4. Run %APPL_TOP%envshell.cmd to create a dos command prompt window with
the required environment settings for Oracle Applications 11i.
Christina M. Zanoni Page 5 4/5/2012
6. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
5. Within the DOS window, CD to the patch stage directory and run adpatch. Follow
the detailed instructions listed in the readme.txt file for the patch.
Quick overview of Metalink
Metalink, http://metalink.oracle.com, is Oracle’s web based support. Metalink contains
notes and documents that will assist you in successfully solving any issue during an
upgrade. Also, you are able to submit, update, and close TARs online, and search
published bugs in the Oracle Bug database. Metalink search tips and suggestions on
submitting an iTar are discussed later in this paper.
Recommended Reading
As with any upgrade, there are various reference manuals and documents that you should
read to help you prepare for an upgrade. Below are the manuals and notes that we used to
upgrade to Applications 11i.
1. Oracle Applications Installation Manual – this manual contains pertinent
information on Rapid Install that applies to any type of Applications install or
upgrade.
2. Oracle Applications Upgrade Manual – this manual contains the category
upgrade steps that need to be performed to upgrade your installation of Oracle
Applications. You should read this manual from cover to cover prior to beginning
your upgrade.
3. Maintaining Oracle Applications Manual – This manual discusses in detail the
applications utilities that are used throughout the upgrade. A few of the utilities
are listed below.
a. ADADMIN – AD Administration performs database and file system tasks
to ensure that Oracle Applications runs properly. The database tasks
include: validate APPS schema, compile APPS schema, recreate grants
and synonyms for APPS schema, compile flexfield data in AOL tables,
maintain multi-lingual tables, and other database tasks. The file system
tasks include: create Applications environment file, relink Applications
programs, copy files to destinations, verify files necessary for runtime,
generate message files, generate form files, generate report files, generate
graphics files, and generate product jar files.
b. ADPATCH – AutoPatch is a utility used to apply maintenance and
individual patches and mini packs.
c. ADCTRL – AD Controller can assist you in determining the status of
Auto Patch Workers, AD Administration and AutoUpgrade. It is also be
used to restart failed workers.
d. ADAIMGR – This is the AD utility used during AutoUpgrade that will
upgrade your Applications database to Applications 11.5.0.
4. Applications Upgrade Issues Document on Metalink –You will find
information on what pre-upgrade patches to apply. It also lists some solutions to
issues you may have during the upgrade.
5. Oracle Applications Release 11i Notes – This document states the current Rapid
Install patch, consolidated upgrade patches, and corrections or modifications to
the Upgrade/Installation steps.
Christina M. Zanoni Page 6 4/5/2012
7. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
6. Oracle Applications Installation Update Notes – Discusses platform-specific
updates to installing Oracle Applications.
Performing the Upgrade – One Method
There are various methods to performing a successful Oracle Applications 11i Upgrade.
We found the upgrade method discussed below to work effectively in our environment.
We performed the category upgrade steps in the Oracle Applications Upgrade Manual
to upgrade our Oracle Applications 10.7NCA environment to Release 11i. We did not
have to perform all category upgrade steps since ALAS’ installation has minimal
customization. Many of the suggestions listed in this paper are taken from the upgrade
manual.
The steps listed are in the order performed during the actual upgrade. This section may
not list all the necessary steps to upgrade your system. Category one and two steps are
listed in summary in this paper, since we did not have to complete a majority of the steps.
The database upgrade and category one through three steps were performed on the
Applications 10.7NCA/Oracle8i database server. AutoUpgrade and the remaining
upgrade steps were performed on the Applications 11i server. (Note: The “i:” drive
referred in this document is the drive letter of our Applications 11i server.)
Upgrade Database from Oracle 7.3.4 to 8.1.6
The steps discussed below were performed on the Oracle Applications
10.7NCA/Oracle8i database server. This server was a clone of our Applications
10.7NCA/Oracle 7.3.4 server. We found that we were able to trim some time off of the
upgrade if we upgraded the Oracle 7.3.4 database to Oracle 8.1.6 prior to starting the
category upgrade steps. There are two possible methods to upgrading the database:
migration or full export/import. The only option for us was the export/import method
since we had to increase the db_block_size from 2k to 8k. Oracle Applications 11i
requires the db_block_size to be at least 8k.
1. Perform a full export of the database on the Applications 10.7NCA/Oracle 7.3.4
database server, and delete the OracleServiceSID NT service. Make sure that you
set the export parameter “compress=y” or you may encounter problems importing
into the new Oracle 8.1.6 database.
2. On the Applications 10.7NCA/Oracle8i database server, install Oracle 8.1.6.0
Enterprise Edition in a different directory for server partition mode and apply the
Oracle 8.1.6.1.1 patch set as stated in the Oracle Applications 11i Upgrade
Manual. Also, we applied the Oracle 8.1.6.1.5 patch set to bring the Applications
database instance up to the most current 8.1.6.1 patch set.
3. Create the Oracle 8.1.6 instance and applications data files on the Applications
10.7NCA/Oracle8i database server, using the newly created initsid.ora file that
includes the change to the DB_BLOCK_SIZE and removal of Oracle 7.3.4
obsolete parameters.
4. Import the Oracle 7.3.4 export into the newly created Oracle 8.1.6 database. We
set all the tablespaces to “autoexend on” so that there would not be any size issue
during the import.
Christina M. Zanoni Page 7 4/5/2012
8. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
5. After the import, run the SQL script adprepdb.sql supplied with the
Interoperability Patch (patch# 499648). This script runs Oracle Applications
database scripts and Oracle 8.1.6 Enterprise Edition scripts that prepare the
database for compatibility with the Oracle 8.1.6 Enterprise Edition Release.
6. After running adprepdb.sql, configure the NET8 Listener to listen for the Oracle
8.1.6 database and configure the SQL*Net client to allow the Oracle Applications
Release 10.7 file system to connect to the Oracle8i database.
7. Once you have completed these steps, apply the applications interoperability
patches to the 10.7NCA/7.3.4 environment that are supplied on the Upgrade
Preparation Patches CD. Afterwards, apply the applications patches listed in
Metalink Note: 112659.1 “Running 10.7 or 11.0.x against 8.1.6”. You will need
to set your applications environment in a command prompt and use the adpatch
utility to apply the application patches listed above.
8. Compiling Invalid Objects is the final step for your Oracle Applications
10.7NCA7.3.4 environment to work with the Oracle 8.1.6 database. Run a
command prompt, set your applications environment, and run the adadmin utility.
Select the Maintain Applications database Objects Compile Apps Schema
menu option to compile objects owned by the APPS schema.
9. Once this is complete, run %AD_TOP%sqladcompsc.pls to compile invalid
objects owned by other schemas.
You should now be able to begin Category one through three steps on the
Applications 10.7NCA/Oracle8i database server. The purpose of this server was to
allow us to easily repeat the database upgrade and category one through three steps by
replacing the data files and re-upgrading the database.
Installation of Release Maintenance Tools for Applications 11i
(Windows NT and Windows 2000 only!)
The following tools must be installed on all Applications 11i Servers (database, forms,
etc) in order to perform release maintenance: Microsoft Visual C++ version 6.0 + Service
Pack 3 or higher, MKS Toolkit version 6.1a or higher, gnu make (Shareware) version
3.79 or higher. Do not install these tools under the Oracle APPL_TOP directory or in a
directory with spaces in its name. This will avoid issues with relinking of executables.
Verify that each of the executables is in the path after installation and that there is a
gnumake.exe file.
Perform Category 1-3 steps on Apps 10.7NCA/Oracle8i Server
Note: We applied all Applications patches listed in the Upgrade Manual and in the
Release Notes for Applications 11i, since all products are installed under the
Applications 11i environment.
Category 1 steps are upgrade preparation steps that do not require the new software.
These steps are considered pre-upgrade steps since they allow you to continue to use the
current Applications environment. The database upgrade steps include renaming
Applications custom database objects so that they are not dropped during the upgrade
Christina M. Zanoni Page 8 4/5/2012
9. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
process, upgrading to Oracle 8i, and gathering database statistics for the Oracle optimizer
using cost based optimization (CBO). Our Oracle Applications installation had very little
customization, so no action needed to be taken for the System Administration,
Application Object Library, and Applications specific tasks except for applying patches
to Oracle Projects.
Run Rapid Install to lay down the application file structure and run Autopatch in
pre-install mode to apply the most current AD and family consolidated patches on
new Oracle Applications 11i server.
The steps below were performed on the Applications 11i server. We did not run Rapid
Install on the Apps 10.7NCA/Oracle8i server. We copied the needed directories from the
Applications 11i server to the Applications 10.7NCA/Oracle8i server in order to
complete category two and three steps on the Apps 10.7NCA/Oracle8i server.
1. Download the most recent patch from Metalink for Oracle Applications 11i Rapid
Install Wizard. The patch number can be found in the Oracle Applications
Release Notes Release 11i document.
2. Stage the “Start Here” CD to apply the RapidWiz patch using the following steps.
a. Insert the “Start Here” CD into the CD-ROM
b. Create a staging directory on the server (e.g. c:stage11iredcddisk1)
c. Run the command xcopy e d: to copy the contents of the CD
d. Delete the rapidwiz sub-directory from the staging area on the server
e. Apply the RapidWiz patch.
3. Run the command c:stage11iredcd11irapidwiz.cmd to start the installation
process of Applications 11i. Figure 4 below is the first Rapid Install screen that
pops up for you to specify the type of installation.
FIGURE 4
4. The options selected for the upgrade were Install Oracle Applications, Single
Node, Create One Instance, Select the same products for all applications
instances, Character Set: WE8ISO8859P1. The remaining prompts will lead you
through the rest of the install. We determined that selecting “Install Oracle
Applications” installs all the necessary components to complete the upgrade to
Oracle Applications 11.5.3.
5. The next step is to apply the most current AD and family consolidated patches in
pre-install mode to the newly installed Applications 11i server.
Christina M. Zanoni Page 9 4/5/2012
10. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
a. Shut down the Forms, Web, Concurrent Manager, and Applications
Listener NT Services.
b. Run %APPL_TOP%appsora.cmd from the command prompt to set the
Applications 11i environment settings.
c. CD to the patch directory and run the adpatch utility with the command
adpatch preinstall=y.
d. Answer the prompts:
i. When prompted for the Applications Environment Name, enter the
SID.
ii. Type Yes to all for Installed product questions since all products
are installed on the server.
iii. Type the APPL_TOP name. Document the APPL_TOP name you
define in your installation, because you will need it later on in the
upgrade.
iv. Type the copy driver name.
v. These steps will need to be repeated for each consolidated patch.
6. Next, copy the %APPL_TOP%adminpreupg directory from the new
Applications 11i Server to the Applications 10.7NCA/Oracle8i server to prepare
for Category two and three steps.
Category 2 steps are upgrade preparation steps that should be run after unloading the
files from the new Applications software. You can continue using your current version of
Applications after completing these steps. As stated in the above paragraph, we copied
the %APPL_TOP%adminpreupg directory and files from the Apps 11i server to the
Applications 10.7NCA/Oracle8i server.
Category two database upgrade steps include validating the APPS schema.
1. Open a command prompt.
2. Execute %APPL_TOP%fin107.bat file to set the applications environment.
3. CD to the %APPL_TOP%adminpreupg directory.
4. Run the following statement:
sqlplus system/”system’s password” @advrf107.sql APPS APPLSYS.
You will need to review the apps.lst file and correct any issues that are listed in
the file.
The next steps modify the tablespaces to function properly with the new version of
Applications. This includes converting the tablespaces from dictionary managed to
locally managed tablespaces, creating tablespaces for new products and resizing existing
tablespaces. (Note: We created two new tablespaces in the database, a data and an index
tablespace, for the new products that we do not use. This prevented us from having to
administer over 100 data files for the Applications 11i database.)
The remaining database upgrade steps include running sqlplus scripts that assist you in
dropping custom schemas that match “APPS%”, verifying custom index privileges, and
dropping conflicting custom public synonyms.
Christina M. Zanoni Page 10 4/5/2012
11. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
The System Administration tasks for category two steps include deleting unsuccessful
login data, restricting access to the administration directory on the new Applications 11i
server, and purging old concurrent requests. The remaining tasks are module-specific and
may or may not have to be completed depending on your configuration.
Please refer to the Applications Upgrade Manual for a detailed explanation for
completing category two steps.
Category 3 steps prepare your Applications products for an upgrade. Once you begin
Category three steps, all users must log off and will be unable to use Oracle Applications
until the upgrade has successfully completed. Unlike Category 1 and Category 2,
Category 3 steps begin with module specific tasks and end with the System
Administration and Database upgrade steps. The applications patches listed in the
module-specific tasks drop database objects that will cause issues during AutoUpgrade.
We had to perform a majority of the tasks listed in Category 3.
Category 3 System Administration tasks should be completed under the old APPL_TOP.
They include running a SQL script to identify potential ORACLE schema conflicts. You
will need to contact Oracle Support if any issues are discovered during this step. We
completed these steps prior to upgrading the database from Oracle 7.3.4 to Oracle 8.1.6.
The next step includes enabling the SYSADMIN user and password.
1. Connect to Applications 10.7NCA with System Administrator responsibility.
2. Navigate to Security -> User -> Define and query for user name SYSADMIN.
Also, the System Administrator will need to verify that all concurrent requests are
complete.
1. Connect to Applications 10.7NCA with System Administrator responsibility.
2. Navigate to Concurrent -> Requests in the Find Requests window.
3. Select All my requests and click find.
Finally, the System Administrator will need to disable AOL Audit Trail features.
1. Connect to Applications 10.7NCA with System Administrator responsibility.
2. Navigate to Security ->Audit Trail ->Groups, and set the Group State field to
Disable in the Audit Groups window.
The Category 3 database upgrade tasks are outlined below.
1. Shut down all applications listeners and concurrent managers – From the old
APPL_TOP, log in to Oracle Applications as System Administrator, choose
Administer Concurrent Managers, navigate to the Control field and select
Deactivate. The Applications forms and web listeners can be shutdown from
Control Panel -> Services.
2. Migrate or upgrade to Oracle8i Enterprise Edition – In addition to the steps
completed prior to starting Category one steps…
a. Set the following init.ora parameters
i. optimizer_mode = choose
Christina M. Zanoni Page 11 4/5/2012
12. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
ii. _sqlexec_progression_cost=0
iii. db_name=”SID”
iv. processes=175
b. Dictionary-managed tablespaces will need to be migrated to locally-
managed if it was not performed in Category 2, Step 2 Database Upgrade
Tasks.
3. Set up Net8 on the database server – This was completed prior to starting
Category one steps.
4. Run preparatory scripts on the database server – According to the Release Notes
for Apps 11.5.3, the database preparatory scripts from patch 1252938 should be
used for this step. However, these files did not work on our Apps
10.7NCA/Oracle8i server, since they reference the directory structure created by
the Apps 11i installation. The only differences between the files were directory
locations. So, we opted to use the pre-patched version of the files. Also, we had to
copy certain directories from the Apps 11i server to the Apps 10.7NCA/Oracle8i
server to successfully complete this step. This was a work-around to having to
install the complete Apps 11i file structure on the Apps 10.7NCA/Oracle8i server.
“Object exists” error messages can be ignored when running these scripts.
5. Gather database statistics for CBO – This step is a repeat of the Category 1, Step
5, of the Database Upgrade steps. The only variation was that we copied the
%APPL_TOP%fnd6.1patchsc107sqlafstatrn.sql file in the old APPL_TOP
directory to the %ORACLE_HOME%appsadmin directory for Oracle 8.1.6. The
steps to gather database statistics for CBO are listed below.
a. Open a command prompt and set your applications environment.
b. CD to %ORACLE_HOME%appsadmin and run the command:
sqlplus apps/”apps password” @afstatrn.sql FALSE.
c. If you receive the “end-of-file on communication channel” error rerun the
following command until the CBO statistics gathering has completed:
sqlplus apps/”apps password” @afstatrn.sql TRUE.
d. Back up your data files prior to the next step.
6. Install database objects for the Oracle HTTP server – We had to perform the
following additional steps on the cloned Apps 10.7NCA/Oracle8i server due to
the fact that we did not install the Applications 11i file structure on the
Applications 10.7NCA/Oracle8i server.
a. Create directory %ORACLE_HOME%apachemodplsql.
b. Copy the iAS ORACLE_HOME directory from the Apps 11i server to the
cloned Apps 10.7NCA server directory created in step a.
c. Copy the %APPL_TOP%adminadmodpls.sql from the Apps 11i server
to the %ORACLE_HOME%appsadmin directory on the Apps 10.7NCA/
Oracle8i server.
d. Run the command:
sqlplus sys/”sys password” @admodpls.sql “SYS PASSWORD” “DEFAULT
TABLESPACE for the public web applications schema” “Temporary
Tablespace” “Net8 Connect String required to connect to your Applications
Database”.
Christina M. Zanoni Page 12 4/5/2012
13. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
e. Review the *.lst file. The only issue for us was that the admodpls.sql file
calls docload.sql, which does not exist on any server. NT does not use this
file.
Gather database information – The adupinfo.sql script gathers
information that will be pertinent to steps performed later in the upgrade. CD
to %APPL_TOP%adminpreupg and run the following command:
sqlplus apps/”APPS password” @adupinfo
7. Follow these steps to check SQL*Plus options:
a. Log on to SQL*Plus.
b. Type show pause (confirm its OFF).
c. Type show newpage (confirm its anything but 0).
8. Verify Rollback segment Sizing – You will need to verify that your rollback
segments are large enough to avoid the ORA-1555 “Snapshot too old” error. The
Upgrade manual recommends having one segment per worker with an extent size
of one MB and a minimum size of 60 MB. We followed the recommendation with
success.
9. Turn off automatic archive logging – Automatic archive logging is not set in our
environment. If your database is set to automatic archive logging, follow the
instruction for this step in the Upgrade Manual.
10. Disable custom, triggers, constraints, and indexes – These objects may cause the
upgrade to fail if you do not disable them.
11. Run preparatory scripts on the admin server – Check the spool file for errors. You
may come across some invalid packages, which is acceptable since they should be
validated during AutoUpgrade. The steps below outline how we completed this
step on the Apps 10.7NCA/Oracle8i server.
a. In a new command prompt, set ORACLE_SID=”SID”,
ORACLE_HOME=”directory of oracle 8.1.6 installation”.
b. Copy all %APPL_TOP%admin*.sql scripts on the Apps 11i Server to the
same directory structure on the Apps 10.7NCA/Oracle8i Server.
c. From the directory above on the Apps 10.7NCA/Oracle8i Server, run
sqlplus apps/”apps password” @adsysapp.sql “System’s password”
d. Check adsysapp.lst file for errors.
12. Back Up the Oracle Applications database and servers – This will allow you to
start at this point if AutoUpgrade fails in a miserable way.
13. Complete upgrade steps for NLS databases – We did not have to perform this step
due to the fact English is the only language used in the database.
Run AutoUpgrade to upgrade Oracle Applications to Release 11.5.0
AutoUpgrade is completed on the Apps 11i Server. Prior to running the AutoUpgrade
utility (adaimgr), you will need to copy over the database files from the Apps 10.7NCA/
Oracle8i server to the Apps 11i server. The steps below outline one method of
completing this process.
1. The initsid.ora file created by Rapid Install was used for the database. Below is a
list of items we modified in the file created by Rapid Install.
a. Location/name of control files
b. Rollback segment names
Christina M. Zanoni Page 13 4/5/2012
14. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
c. Uncommented JAVA_POOL_SIZE.
2. On the Apps 10.7NCA/Oracle8i server run the following commands to assist
you in moving the database over to the new Apps 11i Server:
a. In a Command window, log into server manager as internal on the Apps
10.7NCA/Oracle8i Server.
b. Type command: Alter database backup controlfile to trace;
c. Copy the most recently created trace file TRCxxxx.trc from the
%ORACLE_HOME%RDBMSTrace directory into the new
Applications 11i Server.
d. Edit this trace file to prepare it for execution:
i. Rename it to Sid.trc.
ii. Edit the text to eliminate all lines before the #-commented lines.
iii. Change the startup command to: STARTUP NOMOUNT
pfile=initsid.ora.
iv. Change the create command to CREATE CONTROLFILE SET
DATABASE "SID" RESETLOGS.
v. Edit the last lines of the file to:
1. #RECOVER DATABASE
2. # Database can now be opened normally.
3. ALTER DATABASE OPEN RESETLOGS;
3. Copy the data files to the new Apps 11i Server. Do not copy the log or control
files, since they will be created from the trace file described in step 2.
4. Delete the OracleServiceSID service and recreate it with the proper password for
SYS.
5. Log into svrmgrl as INTERNAL and set your applications environment.
6. Run @sid.trc file created in step 2.
An issue related to running all steps prior to AutoUpgrade on the Apps
10.7NCA/Oracle8i server was that we had to rerun Category 3 Step 4 steps to recreate
CTX on the new Apps 11i Server. The steps to do so are outlined below:
1. Copy %ORACLE_HOME%apps directory on the Apps 10.7NCA/Oracle8i
server to 8.1.6 ORACLE_HOME on the Apps 11i Server,
2. Open a command prompt on the Apps 11i Server
a. set ORACLE_SID=SID
b. set ORACLE_HOME=i:Oraclesiddb8.1.6
c. run sqlplus as ctxsys user and drop library dr$lib and library dr$libx.
These objects contain hardcoded directory structures that reference the
cloned Apps 10.7NCA server directory layout.
3. Run the following statement in the “8.1.6 ORACLE_HOME”AppsAdmin
directory from a command prompt. The SQL statement recreates the libraries
dropped above:
sqlplus internal/”internal’s password” @addbctx.sql <remove context:FALSE>
<Default Tablespace:CTXSYS> <Temporary Tablespace: TEMP > <Intermedia
Shared Library: “8.1.6 ORACLE_HOME”binoractxx8.dll on Windows NT>
4. Check to see if Intermedia is set up properly by running the following statements
as the CTXSYS user in SQLPLUS. There should be no errors.
create table test (nr number primary key, test_text varchar2(500));
Christina M. Zanoni Page 14 4/5/2012
15. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
insert into test values (1,'This is a test');
insert into test values (2,'of the Intermedia text installation');
commit;
create index test_idx on test(test_text) indextype is ctxsys.context;
You will need to read the Maintaining Oracle Applications manual prior to executing
AutoUpgrade. The AutoUpgrade chapter defines AutoUpgrade thoroughly and assists
you in responding to the AutoUpgrade prompts. The steps we performed to begin
AutoUpgrade are outlined below. AutoUpgrade will take anywhere from 8-12 hours to
complete with no errors.
1. Open a command prompt and run %APPL_TOP%appsora.cmd to set
applications environment.
2. run adaimgr
a. We did not modify items listed in Menu Options 1-2.
b. Menu Option 3- Choose Data Parameters
New Products Settings – 0 for sizing, APPLSYSD for MAIN and
DEFAULT tablespaces, and APPLSYSX for INDEX Tablespaces.
These were the two tablespaces created during the Category 2
upgrade steps.
Post-Upgrade Steps
Post-upgrade steps are performed after the completion of AutoUpgrade. These steps
include applying a patch to bring your database to the full Oracle Applications 11.5.3
release level, steps to complete prior to using Oracle Applications (Category 4), steps to
complete prior to using specific Oracle Applications products (Category 5), and steps to
complete prior to using a certain feature in Oracle Applications (Category 6).
Moving from Applications 11.5.0 to Applications 11.5.3
Before you begin Category 4 steps, you must bring your database to the full Oracle
Applications 11.5.3 release level by applying a patch to move from Applications 11.5.0
to Applications 11.5.3. The steps to complete this process are outlined below:
1. Open a command prompt and run the %APPL_TOP%appsora.cmd file to set the
applications environment.
2. CD to %AU_TOP%patch115driver.
3. Run ADPATCH using the d_107_to_1153.drv database driver provided with
patch 1642796. The copy driver of this patch was applied prior to AutoUpgrade in
pre-install mode during the application of the most current AD and family
consolidated patches on the new Oracle Applications 11i server. The database
driver will take anywhere from 11 to 13 hours to complete with no issues.
Category 4 includes steps that are performed against your database and to individual
products. The complete Oracle Applications system is affected, and all steps must be
completed prior to any user logging in and attempting to use any portion of the
application.
Step 1 of the Database Upgrade Steps, “Install NLS translated software”, was omitted
because American English is the only active language in our installation. Database
Christina M. Zanoni Page 15 4/5/2012
16. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
Upgrade Steps 2-4 were reordered as recommended by Oracle Applications Release
Notes Release 11i(11.5.3).
Step 2 is to run Rapid Install to configure and start server processes. The steps we
performed to complete this step are listed below.
1. Verify that all Applications NT Services are running through Control Panel
Services.
2. In command prompt, run %APPL_TOP%appsora.cmd.
3. After establishing environment, run
C:stage11iredcddisk1rapidwizrapidwiz.cmd.
4. Once the Oracle Applications Rapid Wiz Installer opens, choose Configure
Existing Applications Instance.
5. Next screen, choose open an existing configuration file.
6. Our file was located at c:upgradeconfig.txt.
7. Verify that all passwords are correct as stated in Rapid Install Screen.
8. Next screen, choose ... Install all products on this server...
9. Insert Rapid Install ApplTop Disk 1 into drive and continue.
10. The Oracle Applications Rapid Install Wizard begins to run and this should
complete in about 15 minutes.
11. Verify that the Concurrent Manager for SID has been created. If not follow the
steps below:
a. In a command prompt, set the Applications environment by running the
command file %APPL_TOP%appsora.cmd.
b. Navigate to %FND_TOP%bin and type ccmsetup.exe.
c. Use the [add] button to create the service with these settings:
Schema Name: APPS
Schema Password: [APPS PASSWORD]
Confirm Password: [APPS PASSWORD]
TNS Aliasname: SID
d. Click OK button to create the service.
e. Reboot the applications server.
f. Manually start Concurrent Manager from the NT Services Panel. Verify that
the service runs as the NT user that owns Oracle Applications or the service
will not function properly. Figure 5 below is a snapshot of the settings for our
Concurrent Manager service.
Christina M. Zanoni Page 16 4/5/2012
17. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
FIGURE 5
Step 3 is where you reset ORACLE schema passwords. It is recommended to reset
passwords for new products since the default password is the schema name. Maintaining
Oracle Applications discusses the steps to reset Oracle schema passwords.
Step 4 of Category 4 is to gather database statistics for CBO.
1. Run the following statement to gather database statistics for CBO:
sqlplus apps/”apps password” @%ORACLE_HOME%AppsAdminafstatrn.sql FALSE.
You may get the "end-of-file on communication channel" error.
2. Rerun the script if you receive the error with the following parameters until the
CBO statistics gathering has completed:
sqlplus apps/”apps password” @afstatrn.sql TRUE
Category 4, steps 5 through 9, relate to customization to Oracle Applications. We did not
have to complete these steps due to the fact that our installation has very little
customization.
We performed step 10, Re-enable custom triggers, constraints, and indexes, where
trigger name indicates your customized triggers, due to the fact we have a few
customized triggers. The command to perform this step is:
alter trigger trigger name enable;
XML Parser for PL/SQL is installed during step 7 of Category 4. Below are some
modifications and verifications that will need to be done in order to successfully
complete this step.
1. Verify that ORACLE_SID=SID and that LOCAL is not set.
2. modify i:oraclesiddb8.1.6binloadjava.bat as follows:
a. set JAVA_HOME=i:oraclesidcomnutiljre1.1.8
b. set
JRECLASSPATH="i:oraclesiddb8.1.6jdbclibclasses111.zip;i:oraclesiddb8.1.6lib
aurora_client.jar;i:oraclesiddb8.1.6sqljlibtranslator.zip;i:oraclesiddb8.1.6sqljlib
runtime.zip;i:oraclesiddb8.1.6javavmlibaurora.zip;i:oraclesidora8.0.6jdklibclas
ses.zip"
Christina M. Zanoni Page 17 4/5/2012
18. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
c. modify
i:oraclesiddb8.1.6appsutiladminoraclexsu12liboraclexmlsqlload.bat
with the apps username and password (set
USER_PASSWORD="apps/apps password")
Category 4, step 8, relates to Validating and compiling the APPS schema(s).
1. Open a command prompt.
2. Set your applications environment by executing the %APPL_TOP%appsora.cmd
command file.
3. Run the adadmin utility.
4. Select Maintain Applications database ObjectsValidate APPS Schema(s)
option. We applied numerous patches to fix invalid APPS objects during this step.
5. After each patch application. We recompiled the APPS schema to verify that the
patch fixed the invalid object.
Afterwards, there were 28 invalid APPS objects in the database. You should back up your
applications server prior to finishing Category 4 steps.
The remaining post-upgrade steps are system administration or product-specific tasks.
We did not have to complete a majority of the steps due to the fact we use few products
and have very little customization. However, the steps below had to be completed prior to
any user logging into the core Oracle Applications. In our case the URL to log into the
core Applications is: http://server_name:port_number/OA_HTML/US/ICXINDEX.htm
1. Validate SYSADMIN login.
2. Run the following SQL STATEMENT as Apps User to verify SYSADMIN
password:
select fnd_web_sec.validate_login('SYSADMIN','SYSADMIN') from dual;
Result should be “Y”. If not, refer to note 1072328.6: How to Reset only the
SYSADMIN Password on Metalink to assist you in resetting the SYSADMIN
password.
3. Verify APPS_WEB_AGENT profile_option_name by querying
applsys.fnd_profile_option_values table and update value if incorrect. The value
for our configuration was http://servername:port/pls/SID.
4. Grant select on FND_PROFILE_OPTIONS and
FND_PROFILE_OPTION_VALUES owned by APPLSYS TO APPLSYSPUB
and create public synonyms to these tables.
5. Run the following SQL statement:
select server_id, server_address from applsys.fnd_application_servers;
If it returns NO rows, recreate the dbc by modifying file %COMMON_TOP
%installadgendbc.cmd with correct passwords and running the file after setting
the applications environment.
6. Update DAD password for APPS in http://server:port/pls/SID/admin_/
Finishing Your Upgrade
You will need to perform some additional steps to complete the Oracle Applications 11i
upgrade. These steps affect the whole Oracle Applications installation, so they must be
performed in the order specified by the manual after Category 1-6 steps are complete.
Christina M. Zanoni Page 18 4/5/2012
19. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
These steps include dropping obsolete database objects, removing obsolete applications
files, installing online help, recompiling flexfields, and verifying environment setup files.
We did not have to remove any obsolete applications files since the Oracle Applications
11i server was a fresh install.
The final steps describe how to configure the client software to use the Oracle Java
Virtual Machine (JVM) on a web client. The steps include creating a digital signature and
repackaging the Jinitiator. Two issues that we had to resolve are listed below.
1. Modify i:oraclesidoraiASApachejservetcssp_init.txt file -- change ""
directory references to "/"
2. Fix issue in relation to ICX_FORM_LAUNCHER as follows:
a. Log into Core Applications http://server:port/dev60cgi/ifcgi60.exe
b. Log into System Administrator responsibility and navigate to Profile -->
System --> Do a Find for Profile:ICX:Forms Launcher.
c. Set Profile to http://server:port/dev60cgi/ifcgi60.exe.
Documenting the Upgrade Process
It is imperative to document each pre-category, category, and post-category step of the
upgrade in detail. If you do not, there is a good chance that you may overlook a step that
may seem insignificant, but actually affects the success of the upgrade. The amount of
time that each step takes should be part of your documentation. This will assist you in
calculating the estimated time for the upgrade to complete from start to finish.
We were able to complete the production upgrade with few issues and in the time we
stated, since we documented the test upgrade in detail. Also, you should keep a copy of
all Metalink notes that assisted you in solving upgrade problems. These notes are a great
reference to explain how you solved a problem.
Production Applications Upgrade Suggestions
It is extremely important that you follow the exact steps from the test upgrade to prevent
new issues arising during the production upgrade. You may encounter different issues if
you vary from the upgrade steps performed in test. If possible, perform the steps of the
upgrade more than one time in test. This will allow you to become more familiar with the
Applications upgrade process and to discover any issues missed during the initial test
upgrade.
Tips on Solving Upgrade Issues
Metalink Search Tips
The ability to resolve an issue quickly will allow you to continue with an upgrade in a
timely manner. Searching on the oracle error, script name that failed, or the applications
error on Metalink can assist you in solving upgrade issues. We were able to resolve many
issues by following suggested fixes described in Metalink Notes for similar problems.
Below are some examples of how we were able to resolve upgrade issues without the
assistance of Oracle Support.
Christina M. Zanoni Page 19 4/5/2012
20. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
During Auto Upgrade, which is after the Category 3 steps for the Applications 11i
Upgrade, we received an oracle error: ORA-25150 that caused all the workers to fail. We
went to http://metalink.oracle.com, logged in and typed “ORA-25150” to search for a
solution to the oracle error. We received the results listed below in Figure 6.
FIGURE 6
The first document that we reviewed was 11.5 Upgrade Fails on mscpart.odf When
Using Locally Managed Tablespaces, because the error occurred during the 11.5
upgrade and all tablespaces were locally-managed. We learned from this document that
altering extent parameters are not permitted for locally-managed tablespaces with
uniformed extents. The suggested fix for this issue was to run the following command on
each tablespace where the worker failed:
exec dbms_space_admin.tablespace_migrate_from_local('Tablespace Name');
Autoupgrade continued after converting the locally-managed tablespaces back to
dictionary managed tablespaces.
When workers fail on a specific script, you can go to http://metalink.oracle.com and
search on the script name. Workers failed when attempting to upload the file fndlang.dt
while running the following statement:
%FND_TOP%binFNDLOAD APPSAPPS 0 Y UPLOAD_PARTIAL FND_LANGUAGE
UPLOAD_MODE=INSTALL
We were able to solve this issue and restart the workers successfully by reading and
following the steps in the note: FNDLOAD and fndsub05.drv fail with illegal escape
sequence (o) in fndlang.ldt.
Oracle Applications error messages did not occur during the database upgrade steps of
the applications upgrade. Most DBAs will not have to resolve applications errors, since it
is the responsibility of the Applications System Administrator. However, our DBA had to
fill the role of System Administrator due to the fact that our Financials Applications
Administrator was unavailable during much of the upgrade to Oracle Applications 11i.
Below is an example of how we were able to resolve an application issue during the
upgrade.
Christina M. Zanoni Page 20 4/5/2012
21. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
There were applications error messages (APP-668, APP-874) in the log files while
compiling flexfields. We searched Metalink, this time for the following words: App-668
APP-874 flexfield. Various notes on Metalink stated that patch 1303846 should be
applied to fix these errors. We applied the patch and the errors disappeared.
Remember to always backup your database and applications servers prior to applying a
patch. This will allow you to restore the system to its previous state if the patch does not
fix the issue and introduces additional problems.
Opening an iTar with Oracle Support for an Apps upgrade issue
Issues may occur in the upgrade when you have to open an iTar with Oracle Support due
to lack of solutions or information on Metalink. You should supply as much data as
possible when creating an iTar to expedite the resolution of the issue. The information
critical to solving any application upgrade issue is described below. We were able to
resolve a production upgrade issue in one day by supplying all the information below
when opening an iTar.
1. The category step where the issue occurred. This will allow the analyst to
figure out what upgrade steps have been performed to the system prior to the
issue.
2. Any log files documenting the issue. You should submit the following files
when issues occur during AutoUpgrade or a patch application: adpatch.log file
created during a patch application and adwork0x.log created from workers.
3. The name of the script that caused the error and the header information
from the script. The script name and location can be found in the log files. The
header information is located at the top of the script in question and gives the
analyst the version of the script that is causing the issue.
4. A list of all patches applied during the Oracle Applications Upgrade. This
will assist the analyst on deciphering if you are on the most current patch set of a
product or if you will need to apply patches.
Conclusion
The Oracle Applications 11i upgrade was a tedious and rewarding upgrade. There were
many new technologies to learn, and our DBA was able to tone her technical problem
solving skills during the upgrade.
This document is not an “all inclusive” document and should not be used as the only
resource to upgrade Oracle Applications. We cannot guarantee it to be completely error
free, and many methods discussed in this paper may not function properly in another
environment.
If you have any questions about the information in this document, please email Christina
Zanoni at cmzanoni@alas.com.
Christina M. Zanoni Page 21 4/5/2012
22. Paper Number: 707 - Tips on Successfully Upgrading Applications 11i for the Non-Apps DBA
Resources
1. http://metalink.oracle.com
2. http://www.oracle.com
3. Oracle Applications Installation Update Notes Release 11.5.3 for Windows
NT2000
4. Upgrading Oracle Applications Release 11i (11.5.3)
5. Oracle Applications Release Notes Release 11i (11.5.3)
6. Installing Oracle Applications Release 11i (11.5.3)
7. Maintaining Oracle Applications Release 11i (11.5.3)
8. Oracle Applications Concepts Manual 11i (11.5.3)
9. Upgrading from 10.7 to 11i – A Rough Guide - OAUG Spring 2001 White Paper
#121 – written by Milan Rahman
Christina M. Zanoni Page 22 4/5/2012