OPatch supported patch methods in RAC environment

Before starting this article let's have an idea about oracle Interim Patch (formerly known as a "one-off" patch). An oracle Interim Patch or "one-off" patch is a bug fix (or set of fixes) made available to customers to solve a particular bug. This is necessary because for business reasons many ones cannot wait till the next Patch Set or new product release to get a fix. Oracle sometimes recommend interim patches specially security patches to be applied to your systems. Note that , interim patches can only be applied to a particular product version (base release or patch set). That is an interim patch created for 10.2.0.2 should NOT be installed on 10.2.0.3 or 10.2.0.4.

The OPatch (also called the Interim Patch Installer) is the oracle supplied utility which is used to apply interim patches to Oracle's database software. OPatch/Interim Patch Installer supports -
- Applying an interim patch
- Rolling back the application of an interim patch
- Conflict resolution when applying an interim patch after previous interim patches have been applied
- Reporting on installed products and Interim (One-Off) patch

You can download OPatch from metalink as patch 6880880. Patches for Oracle 9i Release 2 Patchset 1 (and beyond) do not include OPatch. OPatch is included from Oracle9i Release 2.

OPatch supports 3 different patch methods on a RAC environment.
1)Patching RAC as a single instance (All-Node Patch)
2)Patching RAC using a minimum down-time strategy (Min. Downtime Patch)
3) Patching RAC using a rolling strategy - No down time (Rolling Patch)

1) Patching RAC as a single instance (All-Node Patch)
In this mode, OPatch applies the patch to the local node first, then propagates the patch to all other nodes, and finally updates the inventory. All instances will be down during the whole patching process.

The flow diagram of All-node patch is,

. Shutdown all Oracle instances on all nodes
. Apply the patch to all nodes
. Bring all nodes up

2) Patching RAC using a minimum down-time strategy (Min. Downtime Patch)
In this mode, OPatch patches the local node, asks users for a sub-set of nodes, which will be the first nodes to be patched. After the initial subset of nodes are patched, Opatch propagates the patch to the other nodes and finally updates the inventory. The downtime would happen between the shutdown of the the second subset of nodes and the startp of the initial subset of nodes patched.

The flow diagram of Minimum downtime is,

. Shutdown the Oracle instance on node 1
. Apply the patch to the Oracle instance on node 1
. Shutdown the Oracle instance on node 2
. Apply the patch to the Oracle instance on node 2
. Shutdown the Oracle instance on node 3
. At this point, instances on nodes 1 and 2 can be brought up
. Apply the patch to the Oracle instance on node 3
. Startup the Oracle instance on node 3

3) Patching RAC using a rolling strategy - No down time (Rolling Patch)
With this method, there is no downtime. Each node would be patched and brought up while all the other nodes are up and running, resulting in no disruption of the system.

The flow diagram of Rolling patch (no downtime) is,

. Shutdown the Oracle instance on node 1
. Apply the patch to the Oracle instance on node 1
. Start the Oracle instance on node 1
. Shutdown the Oracle instance on node 2
. Apply the patch to the Oracle instance on node 2
. Start the Oracle instance on node 2
. Shutdown the Oracle instance on node 3
. Apply the patch to the Oracle instance on node 3
. Start the Oracle instance on node 3

How does OPatch select which method to use?
All oracle one off patches are not rolling patch. Oracle developers determine whether a patch will be rolling patch or not. OPatch also follows an algorithm to determine which methods it will use in RAC environment. The algorithm is,

If (users specify minimize_downtime)
patching mechanism = Min. Downtime
else if (patch is a rolling patch)
patching mechanism = Rolling
else
patching mechanism = All-Node

How to determine if a patch is a "rolling patch" or not?
Based on the oracle database version we can determine whether a patch is a rolling patch or not.

- For oracle version 9i or 10gR1 issue,
$ opatch query -is_rolling

Opatch will ask the patch location and then will inform if the patch is or not a "rolling patch"

- In oracle 10gR2 issue,
$ opatch query -all <patch_location> | grep rolling

For Windows, the following command can be used as grep is not suitable:

> opatch query -all C:\stage\10.2.0.3_Mini_Patches\5731537 | findstr rolling

Related Documents
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed
OPatch failed with error code 74 - CheckActiveFilesAndExecutables failed
How to Download Patchset or Opatch from metalink
List of Patchset number in metalink
SEVERE:OUI-67073: Apply Session failed: ApplySession::processLocal() failed in system

SEVERE:OUI-67073: Apply Session failed: ApplySession::processLocal() failed in system

Problem Description
OPatch apply Patch failed with the following error.
SEVERE:OUI-67073: Apply Session failed: ApplySession::processLocal() failed in system, inventory

From the OPatch logfile I get the following entries,

INFO:Finish backing up system for rollback at Thu Apr 01 12:12:13 EST 2010
INFO:Start the Apply preScript at Thu Apr 01 12:12:13 EST 2010
SEVERE:OUI-67021:SEVERE:OUI-67073: Apply Session failed: ApplySession::processLocal() failed in system, Return Code = 1
INFO:Finish the Apply preScript at Thu Apr 01 12:12:13 EST 2010
INFO:Execution of PRE script failed, with return value = 1

Cause of the Problem
The problem occurred due to oracle bug 6880880. This bug fire if the OPatch version in the $ORACLE_HOME isn't up to date.

Solution of the Problem
Look at the post http://arjudba.blogspot.com/2010/03/severe-oui-67073-applysession-failed.html where the similar error is described.

To solve above bug you have to use the latest released OPatch 10.2, which is available for download from My Oracle Support patch 6880880 by selecting the 10.2.0.0.0 release. After you download that OPatch, install it and then try to apply the patch set again.

Related Documents
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed
OPatch failed with error code 74 - CheckActiveFilesAndExecutables failed
How to Download Patchset or Opatch from metalink
List of Patchset number in metalink

SEVERE: OUI-67073: ApplySession failed: Patch ID is null.

Problem Description
While applying Patch 9119284 - 10.2.0.4.3 Patch Set Update it fails with error message "SEVERE: OUI-67073: ApplySession failed: Patch ID is null" like below.

oracle:/nas/sysadm/oracle/psu/HP/9119284>opatch apply
Invoking OPatch 10.2.0.4.3

Oracle Interim Patch Installer version 10.2.0.4.3
Copyright (c) 2007, Oracle Corporation. All rights reserved.


Oracle Home : /oracledba/app/oracle/product/10.2.0/db
Central Inventory : /oracledba/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.4.3
OUI version : 10.2.0.4.0
OUI location : /oracledba/app/oracle/product/10.2.0/db/oui
Log file location : /oracledba/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2010-03-31_01-33-24AM.log

ApplySession failed: Patch ID is null.
System intact, OPatch will not attempt to restore the system

OPatch failed with error code 73

Here goes the contents of /oracledba/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2010-03-31_01-33-24AM.log

SEVERE:OPatch invoked as follows: 'apply '
INFO:
Oracle Home : /oracledba/app/oracle/product/10.2.0/db
Central Inventory : /oracledba/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.4.3
OUI version : 10.2.0.4.0
OUI location : /oracledba/app/oracle/product/10.2.0/db/oui
Log file location : /oracledba/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2010-03-31_01-21-26AM.log

INFO:Starting ApplySession at Wed Mar 31 01:21:28 EDT 2010
INFO:Starting Apply Session at Wed Mar 31 01:21:28 EDT 2010
SEVERE:OUI-67073:ApplySession failed: Patch ID is null.
INFO:System intact, OPatch will not attempt to restore the system
INFO:Finishing ApplySession at Wed Mar 31 01:21:29 EDT 2010
INFO:Total time spent waiting for user-input is 0 seconds. Finish at Wed Mar 31 01:21:29 EDT 2010
INFO:Stack Description: java.lang.RuntimeException: Patch ID is null.
INFO:StackTrace: oracle.opatch.PatchObject.getPatchID(PatchObject.java:543)
INFO:StackTrace: oracle.opatch.ApplySession.loadAndInitPatchObject(ApplySession.java:1488)
INFO:StackTrace: oracle.opatch.ApplySession.process(ApplySession.java:5236)
INFO:StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:1588)
INFO:StackTrace: oracle.opatch.OPatch.main(OPatch.java:619)

Cause of the Problem
The problem occured because OPatch version in the $ORACLE_HOME isn't up to date. But this Oracle error message seemed bogus to me. According to me at least it show throw message like
" The version of OPatch is too old to support for this patchset ".

Solution of the Problem
In order to apply "Patch 9119284 - 10.2.0.4.3 Patch Set Update" you must use the OPatch 10.2 version 10.2.0.4.7 or later. Oracle recommends to use the latest released OPatch 10.2, which is available for download from My Oracle Support patch 6880880 by selecting the 10.2.0.0.0 release. After you download that OPatch and install it
try to apply the patch set again.

Related Documents
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed
OPatch failed with error code 74 - CheckActiveFilesAndExecutables failed
How to Download Patchset or Opatch from metalink
List of Patchset number in metalink

UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed

Problem Description
When applying a Opatch to a 10.2.0.x installation, applying OPatch failed with the following error:

Running prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:

Following executables are active :
/u01/app/oracle/product/10.2.0/lib/libnmemso.so
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

OPatch failed with error code 73

Cause of the Problem
The problem happened because a process is still using the libnmemso.so library which violates prerequisite check of installing one-off patches or patchsets. To install Opatch or patchsets all processes that use the $ORACLE_HOME being patched MUST be shutdown cleanly, this would include databases, listeners, intelligent agents, database control etc.

Solution of the Problem
A similar type of error and solution is discussed in http://arjudba.blogspot.com/2010/03/opatch-failed-with-error-code-74.html.

To implement the solution, please execute the following steps:

Step 01: Identify the process that is using the library libnmemso.so

$ fuser /u01/app/oracle/product/10.2.0/lib/libnmemso.so

Step 02: Stop that process. libnmemso.so library is used by EM and so often it is necessary to stop the enterprise manager control process.

$ emctl stop

Step 03: After it is stopped apply the patch

Step 04: Restart any processes that were previously stopped.

Related Documents
OPatch failed with error code 74 - CheckActiveFilesAndExecutables failed
How to Download Patchset or Opatch from metalink
List of Patchset number in metalink

OPatch failed with error code 74 - CheckActiveFilesAndExecutables failed

Problem Description
When applying opatch it fails with OPatch failed with error code 74. Further error investigation shows that Prerequisite check "CheckActiveFilesAndExecutables" failed which is displayed below.

oracle:/nas/sysadm/oracle/psu/HP/9119284>opatch apply
Invoking OPatch 10.2.0.4.9

Oracle Interim Patch Installer version 10.2.0.4.9
Copyright (c) 2009, Oracle Corporation. All rights reserved.


Oracle Home : /oracledba/app/oracle/product/10.2.0/db
Central Inventory : /oracledba/app/oracle/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 10.2.0.4.9
OUI version : 10.2.0.4.0
OUI location : /oracledba/app/oracle/product/10.2.0/db/oui
Log file location : /oracledba/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch2010-03-31_02-17-24AM.log

Patch history file: /oracledba/app/oracle/product/10.2.0/db/cfgtoollogs/opatch/opatch_history.txt

ApplySession applying interim patch '9119284' to OH '/oracledba/app/oracle/product/10.2.0/db'
Interim patch 9119284 is a superset of the patch(es) [ 7609058 7609057 7592346 7375617 7375613 7375611 7197583 715 5252 7155251 7155250 7155249 7155248 ] in the Oracle Home
OPatch will rollback the subset patches and apply the given patch.
Execution of 'sh /nas/sysadm/oracle/psu/HP/9119284/custom/scripts/init -apply 9119284 ':
  
Return Code = 0

Running prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:


Following executables are active :
/oracledba/app/oracle/product/10.2.0/db/bin/oracle
ApplySession failed during prerequisite checks: Prerequisite check "CheckActiveFilesAndExecutables" failed.
System intact, OPatch will not attempt to restore the system
--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1) OUI-67620:Interim patch 9119284 is a superset of the patch(es) [ 7609058 7609057 7592346 7375617 7375613 737561 1 7197583 7155252 7155251 7155250 7155249 7155248 ] in the Oracle Home
--------------------------------------------------------------------------------

OPatch failed with error code 74

Cause of the Problem
The Prerequisite check failed because one/more oracle files are active, which violates the requirement of opatch apply.

If there is a RAC environment, then before applying patch shutdown all instances and listeners in the RAC environment.

If this is not a RAC environment, shut down all instances and listeners associated with the Oracle home that you are updating.

As you see in the error message
"Following executables are active :
/oracledba/app/oracle/product/10.2.0/db/bin/oracle"
the problem happened because in one instance oracle binary is still active.

There is also another possible reason of Prerequisite check fail due to oracle bug.

Solution of the problem
Ensure that all database instances/services, processes, listeners are shutdown currently those are being patched. You can check whether any process running or not by issuing query,
$ ps -ef | grep oracle
$ ps -ef | grep tns

If any process running you may want to kill that using kill -9 command.
Note that you might also need to exit from sql*plus client tool if you are connected as sysdba.

sql> exit;

If you see there is no oracle database process then possibly you might hit oracle Bug 6006980.
Workaround to solve this bug
1. Copy fuser executable from /bin to /sbin as root user.
2. Re-run the opatch command as "oracle" user.

Related Documents
How to Download Patchset or Opatch from metalink
List of Patchset number in metalink

TNS-01169: The listener has not recognized the password

Problem Description
oracle:/databridge/db DBGEP> $ lsnrctl status LISTENER

LSNRCTL for HPUX: Version 9.2.0.8.0 - Production on 28-MAR-2010 21:19:33

Copyright (c) 1991, 2006, Oracle Corporation. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ravel.gov)(PORT=1521)))
TNS-01169: The listener has not recognized the password

Cause of the Problem
The TNS-01169 error occurred because security feature is enabled for listener i.e password is set in listener.ora file but lsnrctl command is issued without proper authentication.

Solution of the Problem
After you enable listener password in Oracle 9i, you will now require a password whenever you wish to stop the listener or any other listener actions. However in Oracle database 10g, if you are not logged into the operating system with a privileged account i.e OS user is a member of dba group, you will have to enter a password while doing any operation to listener.

As we see from the message our listener version is 9.2.0.8.0 so follow the following steps to solve the problem.

A) If you remember listener password:
Step 01: Invoke lsnrctl command.
$lsnrctl

Step 02: Set the current_listener to the appropriate listener to which you want to do operation.
LSNRCTL> set current_listener {listener_name_here}

Note that you have to issue "set current_listener {listener_name_here}" if your listener name is not default name that is LISTENER.

Step03: Set password to the password that you previously set for listener.
LSNRCTL> set password {password_here}
or simply you can do it interactively as,

LSNRCTL> set password
Password: {enter_your_password_here}

Step 04: Issue your appropriate command.
LSNRCTL> status
or,
LSNRCTL> stop

B) If you forget listener password:
If password is set in plain text within listener.ora file
If you forget listener password then look for listener.ora and see if password is set in plain text. If it is in plain text then you can retrieve it easily and use that in "set password" command and then do operation as you wanted.

If password is encrypted within listener.ora file
Step 01:
If password is set in encrypted format, check the listener process by issuing,
$ps -ef |grep tns
and note down the process id.

Step 02:
Kill the listener process by,
$kill -9 {process_id}

Step 03:
Remove the line PASSWORDS_{listener_name} from the listener.ora file.

Step 04:
Start the listener
$lsnrctl start {listener_name}
and set the password if you want to set the password again. In order to set the password have a look at, How to set listener password

Related Documents
ORA-12518: TNS:listener could not hand off client connection
The listener supports no services
lsnrctl start fails with TNS-12541, TNS-12560,TNS-00511, Linux Error: 111:
Listener Hangs, Child listener process remains persistence
EM Daemon is not running
ORA-12541: TNS:no listener
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Login to Dbconsole, Authentication failed!null Returned
How to Enable Listener Logging and Tracing

TNS-01150: The address of the specified listener name is incorrect

Problem Description
While start Oracle listener it fails with error "TNS-01150: The address of the specified listener name is incorrect" like below.

E:\Arju>lsnrctl start LISTENER12

LSNRCTL for 32-bit Windows: Version 10.2.0.1.0 - Production on 29-MAR-2010 10:38:21

Copyright (c) 1991, 2005, Oracle.  All rights reserved.

Starting tnslsnr: please wait...

TNSLSNR for 32-bit Windows: Version 10.2.0.1.0 - Production
System parameter file is E:\oracle\product\10.2.0\db_2\network\admin\listener.ora
Log messages written to E:\oracle\product\10.2.0\db_2\network\log\listener12.log
TNS-01150: The address of the specified listener name is incorrect

Listener failed to start. See the error message(s) above...

Problem Investigation
If you open the logfile "Log messages written to E:\oracle\product\10.2.0\db_2\network\log\listener12.log" as suggest in the error message above you will see an entry like,

TNSLSNR for 32-bit Windows: Version 10.2.0.1.0 - Production on 29-MAR-2010 15:51:16

Copyright (c) 1991, 2005, Oracle.  All rights reserved.

System parameter file is E:\oracle\product\10.2.0\db_2\network\admin\listener.ora
Log messages written to E:\oracle\product\10.2.0\db_2\network\log\listener12.log
Trace information written to E:\oracle\product\10.2.0\db_2\network\trace\listener12.trc
Trace level is currently 0

Started with pid=4644
TNS-01150: The address of the specified listener name is incorrect

But unfortunately you no longer will see trace file information in the file "E:\oracle\product\10.2.0\db_2\network\trace\listener12.trc" as written in listener log file unless you have enable listener tracing.

Cause of the Problem
The problem happened because the address on which the listener attempted to listen contains a syntax error or have indention problem.

Solution of the Problem
My listener entry within listener.ora file was like below which cause TNS-01150 error.

LISTENER12 =
(DESCRIPTION = 
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1)) 
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1524))
)

I changed the indentation as,

LISTENER12 =
 (DESCRIPTION = 
  (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1)) 
  (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1524))
 )

Now starting the listener is working perfect.

E:\Arju>lsnrctl start LISTENER12

LSNRCTL for 32-bit Windows: Version 10.2.0.1.0 - Production on 29-MAR-2010 18:48:28

Copyright (c) 1991, 2005, Oracle.  All rights reserved.

Starting tnslsnr: please wait...

TNSLSNR for 32-bit Windows: Version 10.2.0.1.0 - Production
System parameter file is E:\oracle\product\10.2.0\db_2\network\admin\listener.ora
Log messages written to E:\oracle\product\10.2.0\db_2\network\log\listener12.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(PIPENAME=\\.\pipe\EXTPROC1ipc)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1524)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER12
Version                   TNSLSNR for 32-bit Windows: Version 10.2.0.1.0 - Production
Start Date                29-MAR-2010 18:48:29
Uptime                    0 days 0 hr. 0 min. 1 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   E:\oracle\product\10.2.0\db_2\network\admin\listener.ora
Listener Log File         E:\oracle\product\10.2.0\db_2\network\log\listener12.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(PIPENAME=\\.\pipe\EXTPROC1ipc)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1524)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "a" has 1 instance(s).
  Instance "a", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

Note that, a variant of above problem throws following TNS-01150:, NL-00303: if the indention/syntax is not ok.

E:\Arju>lsnrctl stop LISTENER12

LSNRCTL for 32-bit Windows: Version 10.2.0.1.0 - Production on 29-MAR-2010 18:50:58

Copyright (c) 1991, 2005, Oracle.  All rights reserved.

TNS-01150: The address of the specified listener name is incorrect
 NL-00303: syntax error in NV string

So as soon as you hit TNS-01150 look for your Oracle version listener.ora syntax and make sure you have tab starting from second lines in listener entry.

Related Documents
ORA-12518: TNS:listener could not hand off client connection
The listener supports no services
lsnrctl start fails with TNS-12541, TNS-12560,TNS-00511, Linux Error: 111:
Listener Hangs, Child listener process remains persistence
EM Daemon is not running
ORA-12541: TNS:no listener
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Login to Dbconsole, Authentication failed!null Returned
How to Enable Listener Logging and Tracing
How to set oracle listener password