Tuning systems to alleviate decreased month end close performance after upgrading to R12 Subledger Accounting from 11i. Reducing Create Accounting completion time by 97% and reducing archivelog generation by 58%. In R12, the Create Accounting Program, GMFAACCP, would stall our production EBS. We were able to decrease the Create Accounting Program for OPM Production Subledger run-time from an original 22 hours to an average of 30 minutes. This was accomplished by modifying: huge pages, SGA, DBWr processes, subledger processor defaults and indexes. These actions compressed our period close process, saving an entire day.

1. Session ID:
Prepared by:
Technical Modifications to
Compress Period End Close
R12.1.3
Reduce Create Accounting Program
impact after upgrading to R12
Subledger Accounting from 11i
10162
Joshua Johnson
Sr. Oracle DBA
Simmons Prepared Foods Inc.
@jj_dba

2. Abstract
Tuning systems to alleviate decreased month end close
performance after upgrading to R12 Subledger Accounting from
11i. Reducing Create Accounting completion time by 97% and
reducing archive log generation by 58%.
In R12, the Create Accounting Program, GMFAACCP,
would stall our production EBS. We were able to decrease the
Create Accounting Program for OPM Production Subledger run-
time from an original 22 hours to an average of 30 minutes. This
was accomplished by modifying: huge pages, SGA, DBWr
processes, subledger processor defaults and indexes. These
actions compressed our period close process, saving an entire
day.
2

3. Technical Modifications to Compress
Period End Close R12.1.3
Who are we?

4. Presenters
■ Joshua Johnson
▪ Sr. Oracle Apps DBA
— Simmons Prepared Foods, Inc.
■ Ned French
▪ Senior Manager
— Centroid
4

5. 5
• Owned/Managed by Founding Partners Since 1997
• Centroid is one of Oracle’s Top 25 Strategic Partners/Platinum Partner Status
• National technology firm focused on Consulting, Managed Services, Cloud
Services and Resell of Oracle products that span the entire enterprise.
• From Applications to Technology to Infrastructure; Centroid Data Center 100%
Oracle Products: Apps, Tech, HW
• AWARDED: 2014, 2015 Oracle Specialized Partner of the
Year – North America; Engineered Systems
Booth #815

6. Agenda
■ Brief background on our Oracle EBS upgrade from 11i to
R12
▪ Including hardware outline
■ OEM 12c Implementation
■ Server OS Level Modifications – Oracle Enterprise Linux
▪ Huge Pages, Memory Limits
■ Oracle Database Modifications
▪ Asynchronous I/O, SGA, DBWr Processes, Indexes
■ Concurrent Program Modifications
▪ Subledger processor defaults
■ Tuning Results – Test
▪ OWC Session
■ Tuning Results – Production
■ Current State
6

7. Our Oracle EBS Upgrade From 11i to R12
Upgrade from version 11.5.10 to 12.1.3

8. 11.5.10 Upgrade to R12.1.3
▪ Database Server
— OL 5.5 64-bit
— 8 Older non-HyperThreading
CPUs
— 128 GB memory
8
■ Upgrade completed using legacy Dell blade servers
▪ Application Server
— OL 5.5 32-bit
— 8 Older non-
HyperThreading CPUs
— 32 GB memory

9. 11.5.10 Upgrade to R12.1.3
Today marks 1010 days since Go Live!
9
Task Date Time
Upgrade Start 03.JUL.2013 18:00
11i Pre-upgrade Functional Setup Complete 03.JUL.2013 23:59
12.1.1 Patching Complete 04.JUL.2013 22:00
12.1.3 Patching Complete 05.JUL.2013 04:00
Post Upgrade Patching Complete 05.JUL.2013 06:00
Business Owner Testing Complete 06.JUL.2013 17:00
All EBS User Accounts Reactivated 07.JUL.2013 17:00

10. OEM 12c Implementation
Central repository for monitoring Oracle database
instances

11. Painting a Rather Bleak Picture - DB Console
11
Create Accounting 06-JULY-2013

12. Painting a Rather Bleak Picture - DB Console
12
What happened here?

13. OEM 12c Implementation
■ Dbconsole
▪ Not meant to monitor a
complex enterprise level
production environment.
▪ Some servers are not
currently configured to
monitor and log errors.
▪ Unable to create the in-
depth alerts necessary to
proactively monitor an
enterprise system.
■ OEM 12c
▪ Decreased time spent in
reactionary processes
▪ Increased Oracle
availability
▪ Improved system health
and reporting
▪ Increased efficiency in
generating reports for
Oracle during an SR
▪ Increased ability to provide
workload reports for our
Application Developers and
management staff.
13

14. Server OS Level Modifications –
Oracle Enterprise Linux
• HugePages
• Memory Limits

15. Server OS Level Modifications – Oracle
Enterprise Linux
■ Provide reserved, non-
swappable memory space for
our SGA
▪ 44 GB HugePages
▪ 42 GB SGA Target
■ Based on recommendations
from the Oracle
nmr.hugepages script.
HugePages
■ Hard and soft memlock set
to match 44 GB huge
pages
▪ 46,143,488 kilobytes
Memory Limits
15
$ grep -i huge /proc/meminfo
HugePages_Total: 22531
HugePages_Free: 20689
HugePages_Rsvd: 19663
HugePages_Surp: 0
Hugepagesize: 2048 kB
http://docs.oracle.com/cd/E37670_01/E37355/html/ol_config_hugepages.html
/etc/security/limits.conf
vm.nr_hugepages = 22531
/etc/sysctl.conf
soft memlock 46143488
hard memlock 46143488

16. Server OS Level Modifications – Oracle
Enterprise Linux
16http://docs.oracle.com/cd/E37670_01/E37355/html/ol_config_hugepages.html

17. Oracle Database Modifications
• Asynchronous I/O • Temporary Datafiles
• Indexes • DBWr Processes
• Redolog Groups • SGA

18. Oracle Database Modifications
■ 29-JUL-2013
▪ SQL> CREATE INDEX
AP.AP_INVOICE_DISTRIBUTIONS_CT
ON
AP.AP_INVOICE_DISTRIBUTIONS_ALL
(invoice_id, awt_related_id) COMPUTE
STATISTICS;
▪ SQL> CREATE UNIQUE INDEX
XLA.XLA_AE_LINES_CT ON
XLA_AE_LINES(AE_HEADER_ID,
AE_LINE_NUM, APPLICATION_ID,
BUSINESS_CLASS_CODE) COMPUTE
STATISTICS;
Indexes Added
▪ SQL> CREATE INDEX
XLA.XLA_AE_LINES_GT_CT
ON XLA.XLA_AE_LINES_GT
(LEDGER_ID,GAIN_OR_LOSS
_FLAG,
CALCULATE_ACCTD_AMTS_F
LAG, BALANCE_TYPE_CODE)
COMPUTE STATISTICS;
18

19. Oracle Database Modifications:
Temporary Files and Redo Logs Migrated to SSD
■ Implemented 5 redo log
groups
▪ 1 member each
— Increased redo log datafile size
from 512 MB to 5 GB
— Allow for future growth while
reducing overhead from log
switching and checkpoints
■ An XLA Create Accounting run
during peak business hours
would generate up to 8 GB of
logs per minute
Redo Log Groups
■ Implemented 8 datafiles
▪ 8 GB each
■ SSD volume separated
from primary Oracle
database volume
▪ Decreased read/write
overhead on the
production database
storage volume
▪ Separate set of disks
Temporary Datafiles
19

20. Oracle Database Modifications
■ Asynchronous I/O enabled
within the database and the
OS / SAN file system
▪ Necessary for utilizing
multiple database writers
■ Asynchronous mismatch
between server and database
is detrimental to system
performance
Asynchronous I/O Setting
■ Increased database writers
from 1 to 4
▪ Allowed us to distribute the
increased workload across
4 processors
▪ init.ora parameter
DBWr Processes
20
db_writer_processes=4

21. Oracle Database Modifications
■ SGA increased slightly from
38 GB to 42 GB
▪ SGA and PGA configured to
fit within the non-swappable
hugepages.
SGA Settings
21
pga_aggregate_target=2g
sga_target=42g

22. Asynchronous I/O Mismatched
22
Subledger Pre-processor followed by Create Accounting
Program for OPM Production Subledger

23. Asynchronous I/O Mismatched:
Database and Server OS
23

24. Asynchronous I/O Matched:
Database and Server OS
24
■ Multiple Create Accounting Executions 23-DEC-2014

25. Concurrent Program Modifications
• Subledger Processor Defaults
• Program Incompatibilities

26. Concurrent Program Modifications
■ Parallel Processing
▪ 3 Processes
— 9000 unit size
■ This allowed us to evenly
distribute the transaction work
load
▪ Daily transactions generally
fall between 22 and 26
thousand
Subledger Processor Defaults
■ Subledger Create Accounting
program made incompatible
with itself
▪ This prevented multiple
sessions from running
simultaneously
— Forced queuing
Program Incompatibilities
26
SQL> show parameter parallel_degree_limit;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
parallel_degree_limit string CPU

27. Concurrent Program Modifications -
Subledger Processor Defaults
27
■ Navigate to Accounting Setups
General Ledger Super User > Setup > Financials >
Accounting Setup Manager > Accounting Setups
■ Navigate to Processor Defaults
Accounting Setups > Accounting Options:
Corporate Set of Books > Subledger Applications
> Update System Options: Process Manufacturing
Financials

28. Concurrent Program Modifications –
Program Incompatibilities
28
■ Incompatibilities Set
▪ GMFAACCP > Process Manufacturing Financials

29. Concurrent Program Modifications –
Program Incompatibilities
29
Incompatible Programs
Concurrent
Program Application Name Program Name
GMFAACCP
Process Manufacturing
Financials Create Accounting
CSTCRACC Cost Subledger Accounting
Subledger Period Close Exceptions
Report
CSTCRACCRCV
Receiving Subledger
Accounting
Subledger Period Close Exceptions
Report

30. Tuning Results – Test
• Test Hardware
• Cisco Blade – 8 CPUs & 384 GB memory

31. Tuning Results – Test: OWC Session
Cleanup Script Provided By Oracle Support: 23-OCT-2013
Based on these results, it looks very promising to run the OPM Create Accounting
Process as follows:
1. Complete OPM Pre-Processor
2. Complete Create Accounting in draft
3. Run Cleanup Scripts prior to next draft Create Accounting run
31
# of
Transactions
Processes Unit
Size
OPM Pre-Proc
Arc
Generated
Mode CA Archive
Generated
Cleanup
Script Run?
Cleanup
Archive
Generated
CA
Completion
Time (secs)
25883 4 4000 512 MB Draft 2 + GB No N/A 393
25883 4 6000 512 MB Draft 2 + GB No N/A 204
25883 3 9000 512 MB Draft 2 GB Yes 52 GB 190
25883 3 9000 512 MB Draft 2 GB No N/A 188
25883 3 9000 512 MB Final 1.5 GB No N/A 240
23260 3 9000 512 MB Draft 1.5 GB No N/A 188
23260 3 9000 512 MB Draft 1.5 GB Yes 11 GB 155
23260 3 9000 512 MB Draft > 1.5 GB No N/A 172

32. Tuning Results – Test: OWC Session 23-OCT-2013
■ Test data set
▪ 25,883 records
■ Major changes today
▪ XML Publisher update
▪ Cleanup script run prior to subsequent CA run
▪ Processes set to 3 with a unit size of 9000
▪ Scalable Flag created
■ Results – Running CA in draft mode
▪ 52% decrease in archive log generation
— Generating 31 GB of archive logs
▪ Reduced run-time to 25 minutes
32
*Note – Previously mentioned indexes were
already running in production prior to this test.

33. Tuning Results – Test: 24-OCT-2013
■ Test data set
▪ 365,128 records (nearly identical to our 22hr close in July)
■ Major changes today
▪ Hugepages increased to 73728 = 144 GB total
▪ SGA increased to 96 GB
▪ Increased unit size to 10,000
■ Results – Running CA in draft mode
▪ 52% decrease in archive log generation
— Generating 31 GB of archive logs
▪ Reduced run-time, completing in only 25 minutes
33

34. Tuning Results – Production
• Production Hardware
• Dell Blade – 8 CPUs & 128 GB memory

35. Tuning Results – Production: 21-NOV-2013
■ Hard & soft memlock set to
46,143,488
▪ This matches the 44 GB
hugepages
■ SGA set to 42 GB
■ Asynchronous I/O enabled
▪ Database & OS / SAN file
system
■ DBWR processes increased
from 1 to 4
▪ Utilize asynchronous I/O
Changes Implemented
■ SSD drives utilized for
temporary datafiles and redo
logs
■ Redo log changes
▪ Log size increased to 5 GB
▪ Implemented 5 redo log
groups
■ Concurrent Program
▪ Processes set to 4 with a
unit size of 4000
35

36. Tuning Results – Production: 21-NOV-2013
■ Results
▪ Create Accounting – Production completed in 21 minutes
▪ Generated less than 2 GB of archive logs
36
$ grep -i huge /proc/meminfo
HugePages_Total: 22531
HugePages_Free: 2187
HugePages_Rsvd: 1161
HugePages_Surp: 0
Hugepagesize: 2048 kB

37. Tuning Results – Production: FEB-2014
37
18-FEB-2014
24-FEB-2014
25-FEB-2014

38. Production – Current State
• New Production Hardware
• Cisco Blade – 16 CPUs & 384 GB memory

39. Multiple Create Accounting Executions:
24-NOV-2014
39

40. Subledger Pre-Processor Program
40

41. Concurrent Program Timings: 26-JAN-2015
to 02-FEB-2015
■ CSTCRACCRCV - Duration Last Run 2:02
▪ 110 Successfully Completed - Average Run Time 147.39
sec
■ GMFAACCP -- Duration Last End of Month Run 9:27
▪ 502 Successfully Completed - Average Run Time 113.67
sec
■ CSTCRACC - Duration Last Run 57 sec
▪ 204 Successfully Completed - Average Run Time 78.38 sec
41

42. References
■ Oracle Database Administrator's Reference 11g Release 2 for Linux
and UNIX-Based Operating Systems
▪ E10839-23
■ Oracle Database Performance Tuning Guide 11g Release 2
▪ E41573-03
■ Oracle Linux Administrator's Solutions Guide for Release 6
▪ E37355-41 - January 2015
■ Oracle Subledger Accounting Implementation Guide Release 12.1
▪ E13628-04
■ Oracle Database PL/SQL Packages and Types Reference 11g Release 2
▪ E40758-03
■ Oracle Financials Implementation Guide Release 12.1
▪ E13425-05
42

43. Questions?
43

44. Thank You
44
You may complete the session evaluation either
on paper or online via the mobile app
Please complete the session evaluation
We appreciate your feedback and insight
Session ID: 10162