RMAN Backup on RAC fails with ORA-00245: control file backup operation failed

Problem Description
In Oracle database version 11.2g RMAN Backup on RAC database fails with following errors while doing controlfile autobackup.

RMAN-00571: =========================================================== 
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== 
RMAN-00571: =========================================================== 
RMAN-03009: failure of Control File and SPFILE Autobackup command on 
ORA_DISK_1 channel at 301/10/2011 12:10:12 
ORA-00245: control file backup operation failed

While backing up through Sql*plus ORA-00245 error also returned.

alter database backup controlfile to '/oracle/cntl_file' 
ERROR at line 1:
ORA-00245: control file backup operation failed

Cause of the Problem
From Oracle documentation we get following information about the error ORA-00245.

ORA-00245 "control file backup failed; target is likely on a local file system"
// *Cause: Failed to create a control file backup because some process
// signaled an error during backup creation. This is likely caused
// by the backup target being on a local file system so it could not
// be accessed by other instances. It can also be caused by other
// I/O errors to the backup target. Any process of any instance that
// starts a read/write control file transaction must have access
// to the backup control file during backup creation.
// *Action: Check alert files of all instances for further information.

RMAN creates a copy of the control file for read consistency, this is the snapshot controlfile. Due to the changes made to the controlfile backup mechanism in 11gR2 any instances in the cluster may write to the snapshot controlfile. Therefore, the snapshot controlfile file needs to be visible to all instances.

The same happens when a backup of the controlfile is created directly from sqlplus any instance in the cluster may write to the backup controfile file.

In 11gR2 onwards, the controlfile backup happens without holding the control file enqueue. For non-RAC database, this doesn't change anything.

But, for RAC database, the snapshot controlfile location must be in a shared file system that will be accessible from all the nodes. So if you put your snapshot controlfile in local file system and if that file are not accessible to other nodes of the RAC database you will likely face Oracle error ORA-00245.

Solution of the Problem
The snapshot controlfile must be accessible by all nodes of a RAC database. If the snapshot controlfile does not reside in on a shared device this error will raise.

1. Check the existing snapshot controlfile location:

rman> show all;

2. Configure the snapshot controlfile to a shared disk:

rman> CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'shared_disk/snapcf.f'; # default

Eid ul-Azha Eid Mubarak Greeting Cards

To every brothers and sisters EID ul-Azha EID Mubarak. I am attaching some EID Mubarak Greeting cards for you.

EID Ul Adha Mubarak

May the blessings of Allah be with you! Eid ul Azha Mubarak!

Wishing you a Blessed EID!

May Allah bless you with all that is beautiful and pure! Eid Mubarak!

Allah brings happiness and smiles to you and your family

May the blessed occasion of Eid Ul Adha grace you with happiness and fill you home with peace!

How to set time and timezone in Linux

This article should make you clear if you have any problem to check or set time or timezone in your Linux operating system. In your computer you should have two clocks.

1)Battery-backed clock which is always running (the hardware, BIOS, or CMOS clock)
2)Operating system clock (system clock).

The BIOS or CMOS clock is called the hardware clock which is only used to set the operating system clock when your operating system boots.

Once Operating system boots, until you reboot or turn off your system, the operating system clock is the one used to keep track of time.

In case of Linux system you have two choices for hardware clock.
1)Hardware clock as UTC/GMT format.
2)Hardware clock as local time.

The preferred option is to keep in UTC format because then daylight savings can be automatically accounted.

Setting the operating system clock in Linux
In order to check your current date in your system issue "date" statement.
# date
Tue Nov 1 15:23:44 BDT 2011

You can also set system date by "date" command.

For example, if you want to set date as October 31, 11:10pm then type "date 10312310" (note that the time is given in 24 hour notation).

If you wanted to change the year to 2011 as well, type "date 103123102011".

If you wanted to set the seconds as well, type "date 10312310.30" or "date 103123102011.30".

Mind it the format is MMDDHHmmYYYY.ss
where, MM= Month in two digits
DD= Day of the month in two digits
HH= Hours in 24 hour format
mm= Minutes
YYYY= Year in 4 digits.
ss= Seconds after decimal.

Setting the Hardware clock in Linux
Issue hwclock to check what the hardware clock is currently set to.

# hwclock
Tue 01 Nov 2011 03:28:52 PM BDT -0.509255 seconds

If the hardware clock is in UTC format and you want to see the local equivalent date then issue, "/sbin/hwclock --utc"

In order to change the hardware clock first set the system clock as I have just shown above.
And then set the hardware clock to the current system clock by issuing
"/sbin/hwclock --systohc" (or "/sbin/hwclock --systohc --utc" if you are keeping the hardware clock in UTC).

Setting UTC or local time
When Linux operating system boots one of the initialisation scripts run the /sbin/hwclock program to copy the current hardware clock time to the system clock. hwclock will assume the hardware clock is set to local time unless it is run with the --utc switch. Instead of editing the startup script, under Red Hat Linux you should edit the "/etc/sysconfig/clock" file and change the "UTC" line to either "UTC=true" or "UTC=false" as appropriate. Here goes the setting of "/etc/sysconfig/clock" file in my system.

# cat /etc/sysconfig/clock
# The ZONE parameter is only evaluated by system-config-date.
# The timezone of the system is defined by the contents of /etc/localtime.
ZONE="Asia/Dhaka"
UTC=true
ARC=false

If you want hardware clock change to UTC then set UTC=true
If you want hardware clock change to local time then set UTC=false

Set timezone
In order to check your timezone you can simply issue "date" command and you can see timezone information.

# date
Tue Nov 1 17:09:10 BDT 2011

Here timezone is BDT.

The timezone under Linux is set by a symbolic link from /etc/localtime to a file in the /usr/share/zoneinfo directory that corresponds with what timezone you are in. For example, since I'm in Dhaka, Bangladesh, /etc/localtime is a symlink to /usr/share/zoneinfo/Asia/Dhaka.

To set your timezone, issue:

# ln -sf ../usr/share/zoneinfo/your_area/zone /etc/localtime

Replace your_area/zone with something like Asia/Dhaka. Have a look in the directories under /usr/share/zoneinfo to see what timezones are available.

# ls /usr/share/zoneinfo

In a nutshell,

1. Run "date MMDDHHmmYYYY.ss" to set current system time/date

2. To set timezone run /usr/share/zoneinfo/your_area/zone /etc/localtime

3. Issue, "/sbin/hwclock --systohc [--utc]" to set the hardware clock.

4. To check whether hardware clock is set to UTC or local time zone issue "cat /etc/sysconfig/clock"

RMAN-20070 on RESYNC or CROSSCHECK archivelog

Problem Description
After switchover operation standby database now becomes primary database and now database backups, crosscheck archivelog all on new primary database fails with error RMAN-20070.

RMAN> CROSSCHECK archivelog all;

starting full resync of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of crosscheck command at 30/10/2011 12:13:00
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of full resync command on default channel at 30/10/2011 12:13:00
RMAN-20070: invalid log history sequence#

Cause of the Problem
This issue is due to Oracle Bug:4755799 RMAN-20070 ERROR ON PRIMARY AFTER DATAGUARD SWITCHOVER

RMAN-20070 can occur during RMAN resync after a dataguard switchover. Also this bug can appear on using CROSSCHECK as it also implies a resync.
This bug remains in,
9.2.0.4 to 9.2.0.8
10.1 to 10.1.0.4
10.2.0.1 to 10.2.0.2
However, this issue does not affect 10.1.0.5, 10.2.0.3 or higher, and 11g.

Solution of the Problem
Workaround:
Disable the rlh_u2 constraint after connecting to the recovery catalog schema.

$ sqlplus rman/rman@catalog_db
SQL> alter table rlh modify constraint rlh_u2 disable ;

Fix:
- Upgrade to 11g, 10.2.0.3 or 10.1.0.5 (fixed versions)
- Apply patch 4755799 if it is available for your platform and release.