How many companies can afford patching regularly their environments? Patching and maintaining a big amount of Oracle Databases is perceived as complex by most companies. Is there a way to make patching simpler and more controlled? What are the best (and worst) practices for Oracle Home maintenance? What are the challenges of the new release model that will bring us one new major release per year? In this session, we will explain some ideas to improve Oracle Home management and database patching, as well as practical examples of automated environments.

2. Effective Oracle Home Management
with the New Release Model
Ludovico Caldara
Computing Engineer
IT Department, CERN
ludovico.caldara@cern.ch
@ludodba
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
2

3. About Ludovico Caldara
■ 18+ Years DBA (Not Only Oracle)
■ ITOUG co-founder
■ OCP (11g, 12c, MySQL) & OCE
■ Italian living in Switzerland
■ http://www.ludovicocaldara.net
■ @ludodba
■ ludovicocaldara
05.12.2018 3
Effective Oracle Home Management
#UKOUG_Tech18

4. CERN
• CERN - European Laboratory for Particle Physics
• Founded in 1954 by 12 Countries for fundamental
physics research in the post-war Europe
• Today 22 member states and world-wide collaborations
• About ~1100 MCHF yearly budget
• 2’300 CERN personnel
• 10’000 users from 110 countries
405.12.2018
Effective Oracle Home Management
#UKOUG_Tech18

5. Fundamental Research
• What is 95% of the Universe made of?
• Why do particles have mass?
• Why is there no antimatter
left in the Universe?
• What was the Universe like,
just after the "Big Bang"?
505.12.2018
Effective Oracle Home Management
#UKOUG_Tech18

6. Effective Oracle Home Management
#UKOUG_Tech18
6
The Large Hadron Collider (LHC)
Largest machine in the world
27km, 6000+ superconducting magnets
Emptiest place in the solar system
High vacuum inside the magnets
Hottest spot in the galaxy
During Lead ion collisions create temperatures 100 000x hotter than the heart of the sun
Fastest racetrack on Earth
Protons circulate 11245 times/s (99.9999991% the speed of light)

7. Quick overview of the
New Release Model
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
7

8. What does it change?
Jan 18 Apr Jul Oct Jan 19 Apr Jul Oct
RU 18.1.0 18.2.0 18.3.0 18.4.0 18.5.0 18.6.0 18.7.0 18.8.0
RUR1 18.2.1 18.3.1 18.4.1 18.5.1 18.6.1 18.7.1
RUR2 18.2.2 18.3.2 18.4.2 18.5.2 18.6.2
RU 19.1.0 19.2.0 19.3.0 19.4.0
RUR1 19.2.1 19.3.1
RUR2 19.2.2
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
8
2

9. What does it change?
Jan 18 Apr Jul Oct Jan 19 Apr Jul Oct
RU 18.1.0 18.2.0 18.3.0 18.4.0 18.5.0 18.6.0 18.7.0 18.8.0
RUR1 18.2.1 18.3.1 18.4.1 18.5.1 18.6.1 18.7.1
RUR2 18.2.2 18.3.2 18.4.2 18.5.2 18.6.2
RU 19.1.0 19.2.0 19.3.0 19.4.0
RUR1 19.2.1 19.3.1
RUR2 19.2.2
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
9
2

10. What does it change?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
10
Short
Development
cycle
Frequent
Upgrades
And Updates

11. “Patch soon, patch often” vs reality
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
11

12. Frequent patching?
• Time consuming
• Complex
• Risky
• Unnecessary (?)
• Error-prone
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
12

13. How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
13

14. How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
14
18.3.0 18.3.1 18.4.0

15. How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
15
18.3.0 18.3.1 18.4.0
634

16. How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
16
18.3.0 18.3.1 18.4.0
634 1089

17. How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
17
18.3.0 18.3.1 18.4.0
634 1089 1257

18. Which patches should I apply?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
18

19. Brave vs Fearful
12.1 >= 12.2
Effective Oracle Home Management
#UKOUG_Tech18
19
How
do I
feel?
PSU
Bundle
Patch
BraveFearful How
do I
feel?
Release
Update
Revision
Release
Update
BraveFearful
• 05.12.2018

20. Brave vs Fearful
12.1 >= 12.2
Effective Oracle Home Management
#UKOUG_Tech18
20
How
do I
feel?
PSU
Bundle
Patch
BraveFearful How
do I
feel?
Release
Update
Revision
Release
Update
BraveFearful
Ludovico(*) Ludovico(*)
(*) yes, it’s Comic Sans
https://mikedietrichde.com/2017/10/24/differences-psu-bp-ru-rur/
• 05.12.2018

21. Common patching patterns
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
21

22. Common worst practices in maintaining homes
• Installing software by hand
• Different patch levels for Oracle Homes with the same path
• Not tracking the installed patches
• Oracle Home paths hard-coded in the operational scripts
• Not minding about naming conventions
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
22

23. Worst approach (big bang)
1. Stop everything
2. In-place binaries patching
3. Database patching, all at once
4. Start everything
Effective Oracle Home Management
#UKOUG_Tech18
23
UP
UP
UP
UP
DOWN
DOWN
DOWN
DOWN
UP
UP
UP
UP
UP DOWN UP
STOP ALL
START ALLPATCH BINARIES
PATCH DBs
• 05.12.2018

24. Worst approach (big bang)
1. Stop everything
2. In-place binaries patching
3. Database patching, all at once
4. Start everything
Effective Oracle Home Management
#UKOUG_Tech18
2405.12.2018
UP
UP
UP
UP
DOWN
DOWN
DOWN
DOWN
UP
UP
UP
UP
UP DOWN UP
STOP ALL
START ALLPATCH BINARIES
PATCH DBs
No easy
rollback
Long
downtime
Complex
planning

25. Different approach (still bad)
1. Install new binaries manually
2. Patch the new binaries
3. Stop, change OH, patch
databases one by one
4. Decommission old binaries
Effective Oracle Home Management
#UKOUG_Tech18
2505.12.2018
UP
UP
UP
UP
DOWN
DOWN
DOWN
DOWN
UP
UP
UP
UP
PATCH BINARIES STOP, MOVE, PATCH
ONE BY ONE

26. Different approach (still bad)
1. Install new binaries manually
2. Patch the new binaries
3. Stop, change OH, patch
databases one by one
4. Decommission old binaries
Effective Oracle Home Management
#UKOUG_Tech18
2605.12.2018
UP
UP
UP
UP
DOWN
DOWN
DOWN
DOWN
UP
UP
UP
UP
PATCH BINARIES STOP, MOVE, PATCH
ONE BY ONE
Complex to
do manually
Repetitive task
when many servers

27. Automate the installation!
• Automated creation
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
27
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply

28. Automate the installation!
• Automated creation
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
28
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply
https://github.com/MarisElsins/getMOSPatch

29. Automate the installation!
• Automated creation
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
29
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply
https://github.com/MarisElsins/getMOSPatch
https://github.com/ilmarkerm/ansible-oracle-home-mgmt

30. Image repo
Automate the installation!
• Automated creation • Software cloning
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
30
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply
18.3.1
18.4.0
Server
18.3.1

31. Image repo
Automate the installation!
• Automated creation • Software cloning
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
31
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply
18.3.1
18.4.0
Server
18.3.1
RUNDECK
ANSIBLE
RHP
OTHER

32. Less paths
Effective Oracle Home Management
#UKOUG_Tech18
3205.12.2018
18.3.0 18.4.0 18.4.0
+ one-off X
18.5.0
+ one-off X
18.5.0
+ one-off X
+ one-off Y
18.3.0 18.4.0 18.4.0
+ one-off X
18.5.0
+ one-off X
18.5.0
+ one-off X
+ one-off Y
18.4.1 18.4.1
+ one-off Y
GOOD:
BAD:

33. Workarounds vs one-offs
• Sometimes through underscore parameters
• Mitigate the number of one-offs
• Less merge requests
• Document it! (why, when, who)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
33
1
2
3
alter system set "_px_groupby_pushdown"=off
comment='Ludo, 03.05.16: W/A for bug 18499088'
scope=both sid='*‘;

34. Rapid Home provisioning
• It might be the perfect product…
• Requires Lifecycle Management Pack
• The technology stack is complex
(Clusterware, ASM, ACFS, NFS, GNS…)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
34

35. Rapid Home provisioning
• It might be the perfect product…
• Requires Lifecycle Management Pack
• The technology stack is complex
(Clusterware, ASM, ACFS, NFS, GNS…)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
35
Managing Gold Images for Easy Patching &
Upgrading with Oracle RHP
Ian Cookson - Oracle
Room 1A, today 15:20

36. Custom approach
• Simple script
• Golden images
• Solid naming conventions
• I blogged about it!
http://www.ludovicocaldara.net/dba/oh-mgmt-[1..7]/
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
36

37. Implementation at CERN
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
37

38. Naming Conventions
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
38
Central Inventory server 1
<HOME NAME= "OraDB12Home1" LOC="/u01/app/oracle/product/12.2.0.1" TYPE="O" IDX="2">
<HOME NAME= "OraDB12Home2" LOC="/u01/app/oracle/product/12.2.0.1_2" TYPE="O" IDX=“3">
Central Inventory server 2
<HOME NAME= "OraDB12Home1" LOC="/u01/app/oracle/product/12.1.0.2" TYPE="O" IDX=“1">
<HOME NAME= "OraDB12Home2" LOC="/u01/app/oracle/product/12.2.0.1" TYPE="O" IDX="2">

39. Naming Conventions
• Central Inventory contains OH name
• OH name matches OH location
• No dots (.) allowed (inventory limitation)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
39
Central Inventory
<HOME NAME=“18_3_0"
LOC="/u01/app/oracle/product/18_3_0"
TYPE="O" IDX="2">
• 1 to 1 mapping: easy to code
• 18c read-only OH uses the name
as subdir in `orabasehome`:
consistent naming!

40. Golden images
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
40
Golden Images Store
18_3_0.zip
18_4_0.zip
18_3_1.zip
SERVER1
18_3_0.zip
18_3_1.zip
SERVER2
18_3_1.zip
SERVER3
18_4_0.zip
Golden Image Metadata
Name Path
------- -----------------
18_3_0 /store/18_3_0.zip
18_3_1 /store/18_3_1.zip
18_4_0 /store/18_4_0.zip

41. Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
42
SERVER1
18_3_0
SERVER2…
First Install

42. Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
43
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -c -n 18_3_0

43. Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
44
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -i 18_3_018_3_0

44. Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
45
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -i 18_3_0 -n 18_4_0
18_3_0
18_4_0

45. Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
46
18_3_0.zip
SERVER1
18_3_0
SERVER2…
Manual patching
18_3_0
18_4_0

46. Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
47
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -c
18_3_0
18_4_0
18_4_0.zip

47. Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
48
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -i 18_4_0
18_3_0
18_4_0
18_4_0.zip
18_4_0

48. What it actually does
Creation workflow
• Copy the Home
• Clean it up
• Create the zip
• Store the zip in the repo
• Store metadata
Effective Oracle Home Management
#UKOUG_Tech18
4905.12.2018

49. What it actually does
Creation workflow
• Copy the Home
• Clean it up
• Create the zip
• Store the zip in the repo
• Store metadata
Installation workflow
• Security checks
• Create new path
• Unzip from the store
• Attach home
• Relink (opt. with RAC)
• Run setasmgid
• Other stuff (e.g. TNS)
Effective Oracle Home Management
#UKOUG_Tech18
5005.12.2018

50. Effective Oracle Home Management
#UKOUG_Tech18
5105.12.2018
DEMO

51. Further development at CERN
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
52

52. Read-only Oracle Home
$ roohctl –enable
Enabling Read-Only Oracle home.
Update orabasetab file to enable Read-Only Oracle home.
Orabasetab file has been updated successfully.
Create bootstrap directories for Read-Only Oracle home.
Bootstrap directories have been created successfully.
Bootstrap files have been processed successfully.
Read-Only Oracle home has been enabled successfully.
Check the log file /u01/app/oracle/cfgtoollogs/roohctl/roohctl-
180822AM102911.log.
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
53

53. Read-only Oracle Home
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
54
18_3_0
18_4_0
/u01/app/oracle/product/
/u01/app/oracle/product/
$ cat $ORACLE_HOME/install/orabasetab
#orabasetab file is used to track Oracle Home associated with Oracle Base
/u01/app/oracle/product/18_3_0:/u01/app/oracle:18_3_0:N:
ORACLE_HOME
ORACLE_HOME
Read only

54. Read-only Oracle Home
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
55
18_3_0
18_4_0
/u01/app/oracle/product/
/u01/app/oracle/product/
/u01/app/oracle/homes/
$ cat $ORACLE_HOME/install/orabasetab
#orabasetab file is used to track Oracle Home associated with Oracle Base
/u01/app/oracle/product/18_3_0:/u01/app/oracle:18_3_0:N:
ORACLE_HOME
/u01/app/oracle/homes/
ORACLE_HOME
orabasehome
orabasehome
18_3_0
18_4_0
Read only
TNS, init, etc.

55. AGILE INFRASTRUCTURE
SERVER
Insertion in the Agile Infrastructure
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
56
Image Store Scripts
REST client
Oracle Home

56. Grid Infrastructure Installation
• Puppet + Rundeck for automatic GI install
• Patched Grid Infrastructure versions beside the first one
https://www.markusdba.net/?p=294
• Out-of-place patching with 18c independent local-mode automaton
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
57
$ rhpctl move gihome –sourcehome old_path -destinationhome new_path

57. Independent Local-mode Automaton (18c)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
58
# $GI_HOME/srvctl enable volume -volume GHCHKPT -diskgroup mgmt
# $GI_HOME/srvctl enable filesystem -volume GHCHKPT -diskgroup mgmt
# $GI_HOME/srvctl start filesystem -volume GHCHKPT -diskgroup mgmt
$ $GI_HOME/srvctl start rhpserver
Local Resources
---------------------------------------------------------------
ora.MGMT.GHCHKPT.advm
ora.mgmt.ghchkpt.acfs
ora.helper
Cluster Resources
---------------------------------------------------------------
ora.rhpserver

58. $ opatch lspatches
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28256701;TOMCAT RELEASE UPDATE 18.3.0.0.0 (28256701)
28090564;DBWLM RELEASE UPDATE 18.3.0.0.0 (28090564)
28090557;ACFS RELEASE UPDATE 18.3.0.0.0 (28090557)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)
OPatch succeeded.
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
59

59. Independent Local-mode Automaton (18c)
• Damn-easy move of GI HOME
• Automatic resource relocation
• Enable Grid Infrastructure Golden Images
• Patching (apply and rollback) is done automatically
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
60

60. Back to the demo…
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
61

61. Questions?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
62

62. Thank you!
Ludovico Caldara
Computing Engineer
IT Department, CERN
ludovico.caldara@cern.ch
@ludodba
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
63