DBConsole does not start due to TimeZone mismatch

Problem Description
The TZ environmental variable is correctly set to GRNLNDST3 and then while startup dbconsole it fails with following message.
Oracle Enterprise Manager 10g Database Control Release 10.1.0.4
Copyright (c) 1996, 2004 Oracle Corporation. All rights reserved.
Timezone mismatch: The agentTZRegion value (AmericaSao_Paulo) in
/oracle/product/10.1.0/hostname_dbname/sysman/config/emd.properties does not match the current
environment TZ setting(GRNLNDST3).
The agent cannot run with this mismatch.
If America/Sao_Paulo is the correct timezone, set your timezone environment
variable to Sao_Paulo and repeat the 'emctl start agent' operation.
If America/Sao_Paulo is not the correct timezone, make sure that the timezone
in your environment is correct, and then run the following command in your local Oracle Home:
'emctl resetTZ agent'
The output of this command will include detailed instructions to follow, to correct the mismatch.

Cause of the Problem
The timezone GRNLNDST3 is not a supported timezone as it is not listed in v$timezone_names.
Those timezones that are listed in v$timezone_names, BRT, BRST, and LMT do not seem to be supported/do not work correctly on AIX as using then for the value of the TZ environment variable results in the incorrect time being returned from the command: date.

Solution of the Problem
In order to solve this problem use the relevant GMT offset for your timezone.
For example, set the TZ environment variable to GMT-3 (if you are in Brazil, East) and issue the command emctl start dbconsole


$ export TZ=Etc/GMT+3
$ export ORACLE_SID=SID
$cd $ORACLE_HOME/bin
$ ./emctl resetTZ agent
(follow the on-screen instructions)

The agentTZRegion value does not match the current environment TZ setting

Problem Description
While starting the enterprise manager using emctl start dbconsole command it fails with following message.
Timezone mismatch: The agentTZRegion value (Asia/Karachi) in
/u01/app/oracle/product/10.2.0/db_1/localhost.localdomain_testdb/sysman/config/emd.properties
does not match the current environment TZ setting(Asia/Karachi).
The dbconsole cannot run with this mismatch.
If Asia/Karachi is the correct timezone, set your timezone environment variable to Asia/Karachi and repeat the 'emctl start agent' operation.
If Asia/Karachi is not the correct timezone, make sure that the timezone in your environment is correct, and then run the following command in your local Oracle Home: 'emctl resetTZ agent'

Investigation of the Problem
After looking at emdctl.trc files for error details we see following entries.
2011-07-11 12:18:02 Thread-4135859904 ERROR main: nmectl.c: nmectl_validateTZRegion, agentTZoffset =360,and testTZoffset for Asia/Karachi:300 do not match
2011-07-11 12:18:03 Thread-4135859904 ERROR main: nmectl.c: nmectl_validateTZRegion, agentTZoffset =360,and testTZoffset for Asia/Karachi:300 do not match

agentTZoffset=360 (This means 6 hours - This is coming from $TZ set in the environment)
and testTZoffset for Asia/Karachi=300 (This means 5 hours - This is coming from emdbconsole's java tranlsation of the timezone (which is correct)

The $TZ on the Operating System is set to Asia/Karachi. When the 'date' command is run from the command line of the linux machine, it can be seen that the date returned is incorrect by an hour.

eg.

$ export TZ=Asia/Karachi
$ date
Mon July 11 16:10:59 PKST 2011

Note* that the time output refers to "PKST" (Pakistan Summer Time)

Cause of the Problem
The time returned by the OS 'date' command is wrong when $TZ is set to Asia/Karachi - as compared to the correct time given in :-

http://www.timeanddate.com/worldclock/city.html?n=757

Which shows that Karachi should be 5 hours ahead of GMT. It can be seen that the date command is returning 'PKST' which is Pakistan Summer Time and which is 6 hours ahead of GMT:-

$ export TZ=Asia/Karachi
$ date
Mon July 11 16:10:59 PKST 2011

However, there has been no Daylight Savings (Pakistan Summer Time) in Pakistan since 2009. As we notice in the link,

http://www.timeanddate.com/worldclock/timezone.html?n=757&syear=2010

which lists any active Daylight savings changes for this TimeZone. Note that 2009 was the last time Pakistan had PKST.

If a different region is picked which is also 5 hours ahead of GMT (eg. one which is defined in $ORACLE_HOME\sysman\admin\supportedtzs.lst) eg.

# Timezone regions with +05:00 standard offset from GMT
Asia/Aqtobe
Asia/Bishkek
Asia/Karachi
Asia/Tashkent
Asia/Yekaterinburg
Etc/GMT-5
Indian/Chagos

and compare it with the generic Etc/GMT-5 (which means 5 hours ahead of GMT)

export TZ=Asia/Tashkent
date
Mon July 11 15:10:59 PKST 2011

export TZ=Etc/GMT-5
date
Wed May 11 15:32:48 GMT-5 2011

export TZ=Asia/Karachi
date
Wed May 11 16:32:58 PKST 2011

It can be seen that these other regions are setting the time correctly.

In this case, the TZ definition for Asia/Karachi is actually set on the machine:-

$ cd /usr/sbin
$ su root
# ./zdump -v Asia/Karachi|grep 2011

On certain machines the output returned from the zdump command is:-

Asia/Karachi Thu Apr 14 18:59:59 2011 UTC = Thu Apr 14 23:59:59 2011 PKT isdst=0 gmtoff=18000
Asia/Karachi Thu Apr 14 19:00:00 2011 UTC = Fri Apr 15 01:00:00 2011 PKST isdst=1 gmtoff=21600
Asia/Karachi Mon Oct 31 17:59:59 2011 UTC = Mon Oct 31 23:59:59 2011 PKST isdst=1 gmtoff=21600

This is incorrect because as we have seen there is no PKST (Pakistan Summer Time) in 2011 (and has not been since 2009).

However on a different machine:-

$ export TZ=Asia/Karachi
$ date
Wed May 11 13:26:31 PKT 2011

the time is returned correctly as 'PKT'

This is related to the release of the OS. The problem appears on :-

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)

But has not been seen to occur on Oracle Enterprise Linux 5 update 6

Solution of the Problem
From the file $ORACLE_HOME\sysman\admin\supportedtzs.lst check the supported timezone list.

# Timezone regions with +05:00 standard offset from GMT
Asia/Aqtobe
Asia/Bishkek
Asia/Karachi
Asia/Tashkent
Asia/Yekaterinburg
Etc/GMT-5
Indian/Chagos

So an example workaround will be:-

$ export TZ=Etc/GMT-5
$ date

and check that the time returned, is the same as the time in Pakistan. (as per http://www.timeanddate.com/worldclock/city.html?n=757)

Once you set start the dbconsole.
$ emctl start dbconsole

Solution of ORA-600 [4194] or ORA-600 [4193]

Problem Description
ORA-600[4194] or ORA-600[4193] occurs when there is mismatch detected between Redo records and rollback (Undo) records. ORA-600[4194] / ORA-600[4193] error comes with two more arguments within bracket.
Arg [a] indicates Maximum Undo record number in Undo block
Arg [b] indicates Undo record number from Redo block
Sample error is below:
ORA-00600: internal error code, arguments: [4194], [19], [33], [], [], []

Solution of the Problem - Drop the undo tablespace
This solution does not deal with Oracle unsupported parameter like _offline_rollback_segments and @_corrupted_rollback_segments to resolve ora-00600[4193]/[4194].

Step 01:
Startup the database in nomount stage using spfile.
SQL> Startup nomount;

In this case ensure you have started your database using spfile. "show parameter spfile" will display the location of spfile.

Create pfile from spfile.
SQL> Create pfile='/tmp/pfile.ora' from spfile ;

Shutdown the database.
SQL> Shutdown immediate;

Step 02:
Modify the pfile.ora and set Undo_managment=Manual
$vi /tmp/pfile.ora
undo_managment=manual

Start the database in mount mode using modified pfile.
SQL> Startup mount pfile='/tmp/pfile.ora'

Ensure that you have undo_managment initialization parameter set to manual.
SQL> Show parameter undo

SQL> Alter database open ;

Once it open, create a rollback segment.

SQL> Create rollback segment r01 ;

SQL> Alter rollback segment r01 online ;

Create a new undo tablespace

SQL> Create undo tablespace undotbs_new datafile 'datafile_name.dbf' size 100M ;

Drop the old undo tablespace

SQL> Drop tablespace undotbs_old including contents and datafiles

Step 03:

Shutdown the database.
SQL> Shutdown immediate;

Start the database using spfile where it had undo_managment initialization parameter set to AUTO.

SQL> Startup nomount ;

Change the undo_tablespace parameter to use new tablespace.

SQL>Alter system set undo_tablespace=undotbs_new scope=spfile;

SQL> Shutdown immediate ;

SQL> Startup

Check if error is reported.

In case of RAC database instances from Instance which is up and running do the following:

SQL> Create undo tablespace undo_new datafile 'undotbs_new' size 100m ;

SQL> Alter system set undo_tablespace=undo_new sid={instance which has corrupt undo tablespace and is down} ;

Now Startup the Instance which is down

SQL> Startup mount

SQL> Show parameter undo

Should show the new undo tablespace created above

SQL>Alter database open ;

SQL>Drop tablespace undotbs_old including contents and datafiles


If all the Instance is down in the RAC due to this error then following the instruction given for Single instance and create new undo tablespace.