ORA-600 or Data Corruption during database shutdown of RAC instance

Problem Description
Today I want to introduce one terrible Oracle bug that fires in Oracle RAC environment, cause data corruption while shutting down RAC instances. The issue can occur when shutting down Oracle RAC instances using the shutdown mode "normal, transactional, or immediate" and the instance(s) are shutdown in a rolling fashion; i.e. other instances remain to operate normally. The issue does not occur, if the instances are shutdown using "shutdown abort" or the database is shutdown as a whole as a workaround.

Cause of the Problem
Data corruption happens due to Oracle bug. Oracle named this bug as 10205230. The following symbols found in the alert log when you hit this bug.
- ORA-600 [kclchkblk_3]
- ORA-600 [kclwcrs_6]
- ORA-600 [ktubko_1]
- ORA-600 [kcratr_scan_lostwrt]
- ORA-600[3020] on the standby database
And data corruption happens in your database. When one RAC instance is working normally and you shutdown another instance either by any of following commands then this corruption happens.
So avoid these commands.
- shutdown normal
- shutdown transactional
- shutdown immediate

Solution of the Problem
1)Shutdown abort whole database instead of shut down one to avoid corruption:

SQL> alter system checkpoint;

$ srvctl stop database -d -o abort -f

The first command from SQL*plus writes out dirty buffers for all instances to minimize instance recovery; The srvctl command shutdown all instances with abort option and its dependent resources. Shutdown abort completely by-passes the vulnerable code path of the bug.

2) If you need shutdown only one instance or in rolling fashion then instead of shutdown normal/transactional/immediate, you should shutdown each instance with the following 2 commands:
SQL> alter system checkpoint local;
SQL> shutdown abort;

The first command writes out dirty buffers for this instance to minimize instance recovery. Shutdown abort completely by-passes the vulnerable code path of the bug. Instance can also be shutdown abort using "srvctl stop instance -d db_unique_name -i instance_name -o abort".

What to do if corruption already happens
As this bug may cause logical corruption to the redo stream so we need to recover corrupt blocks. I have written a topic regarding Handling block corruption in oracle. Also I have written another topic about how to recover data from corrupt blocks.

The solution can be followings at the end.
- Drop and recreate the corrupt object. This may work for objects like indexes.

- Use DBMS_REPAIR package to repair and skip the corrupted blocks, and recreate the object with create table as select. This may work for objects like tables and partition tables, but may result in data loss.

- Failover to a standby database that has not been affected by the data corruption. This may result in data loss. Flashback or perform point-in-time recovery. This may also result in data loss.

- If a standby database encounters ORA-600[3020] stuck recovery, you can invoke trial recovery to see if there are other corrupt blocks:

SQL> recover automatic standby database allow n corruption test;

Here n can be 1 or any other integer.

One can also use the following command to allow standby recovery to continue by marking the problematic block as corrupt:

SQL> recover automatic standby database allow n corruption;

At the end I can say it is terrible Oracle bug!

Impdp fails with ORA-39002 and ORA-39322 during importing transport tablespaces

Problem Description
Oracle data pump import fails with ORA-39002 and ORA-39322 during importing transport tablespaces. The import command is below.

impdp userid=arju/a
directory=d
dumpfile=trans_tbs.dmp
logfile=transport_tablespace.log
transport_datafiles='+DATA/data01.dbf',
'+DATA/data02.dbf'
EXCLUDE=TABLE_STATISTICS
EXCLUDE=INDEX_STATISTICS
EXCLUDE=STATISTICS

And import logfile is as following.

Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options
ORA-39002: invalid operation
ORA-39322: Cannot use transportable tablespace with timestamp with timezone columns and different timezone version.

Cause of the Problem
When importing using transportable tablespaces, if the Timezone table version is different the job will fail. For example on the database from which export is taken has different timezone then the database to which dump will be imported. This difference allows the job to continue, but the tables that have TSTZ type of columns will not be imported. You can check the timezone version by following query.

Select name,value$ from props$ where name='DST_PRIMARY_TT_VERSION'

In source database,

 NAME                     VALUE$
 ------------------------ ----------
 DST_PRIMARY_TT_VERSION   13

In destination database,

 NAME                     VALUE$
 ------------------------ ----------
 DST_PRIMARY_TT_VERSION   14

In the above example the target database timezone (14) is higher than the source database timezone (13) and Data Pump would fail since it checks if the timezone has changed and whether the dumpfiles contain tables that use "TIMESTAMP WITH TIME ZONE" columns.

Solution of the Problem
No workaround and no solution from Oracle database yet. They say they will fix in 12.1g.

Data pump import fails with ORA-29913 when used PARALLEL and REMAP_SCHEMA

Problem Description
Data pump import operation will fail if both PARALLEL (>1) and REMAP_SCHEMA parameter are used. From the data pump import logfile we see following:

Processing object type TABLE_EXPORT/TABLE/TABLE
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
ORA-31693: Table data object "ARJU"."TEST" failed to load/unload and is being skipped due to error: ORA-29913: error in executing ODCIEXTTABLEOPEN callout

Cause of the Problem
This problem is identified as Oracle Bug 8393456 "Data pump import fails when specifying PARALLEL > 1 and REMAP_SCHEMA". The problem remained in Oracle version 11.2.0.1, 11.1.0.7, 10.2.0.4.

Solution of the Problem
The problem is fixed in Oracle database server patchset 11.2.0.2 and in 11.1.0.7 Patch 34 on Windows Platforms. However if you have other version and facing the bug, workaround you can omit the PARALLEL parameter or simply specify PARALLEL=1.

Real Estate Empire sells Bimini Bay Resort in Bahamas

Whether you are planning for a vacation in the islands of Bahamas or to buy a bimini bay resort, Bimini Bay is a perfect place to meet your needs. Bimini is situated on the west in the district of Bahamas, just 48 nautical miles away from South Florida. If you want to swim with Dolphins, celebrate wedding ceremony, drive scuba, passion for fishing Bimini is great place.

Bimini is divided into South Bimini and North Bimini. North Bimini is about 11 kilometers long and only 200 meters wide. For many years tourists would face accommodation problems while visiting Bimini due to few accommodations other than private homes. But now thanks to Bimni Bay Resort. Bimini Bay Resorts and Marina developed their resort with traditional Bahamian architecture which is comprised of 350 individually-owned existing residences. Many of the resorts are top-of-the-line, luxurious amenities, closer to fishermen village, situated on the Northern tip of the island of North Bimini which is popular by the name of "The Bone Fishing Capital of the World,". The Bimini Bay Resort is located on the closest island to the U.S. Mainland.

The Resort is the central point of the sport fisherman as well as for the broader boating and yachting community. Among the AMENITIES & ACTIVITIES there comes Sabor Restaurant, Childrens Activity Center, shopping village, outdoor pools, Mega Yacht Marina pool with swim-up Bar, tennis courts, Business Center and Wireless Internet, Fitness Center, Non motorized watersports, , Snorkeling and Scuba Charters, Convenience Store and Market and activities like scuba diving, snorkeling and boat tours. You can find some of the Bimini Bay activities in Bimini Activities.

The Bimini Bay Resort offers two separate marinas - Fisherman's Village Marina (136 slips) and the Mega Yacht Marina (96 slips) - both equipped with concrete floating docks. The Mega Yacht Marina can accommodate yachts up to 203 feet. It also features its own Customs & Immigration office allowing guests to tie up and clear customs immediately.

Now let's know about company Real Estate Empire. Real Estate Empire is an innovative leader in property sales and marketing, with experienced agents managing transactions efficiently across a spectrum of prices, and geography. They also consult, design, implement and deliver marketing solutions that lead clients into achieving target goals. Real Estate Empire's agility, client-first approach, robust portfolio, and global experience enable to quickly respond with marketing solutions that transform vision into reality. Beside Bahamas Real Estate, Real Estate Empire also offer for Orlando Condos, Luxury Orlando Condos, Reunion Resort in Florida. Recently, Real Estate Empire launched a new site named http://resortbahamas.com/ which offers Bahamas Homes, Bahamas Resort, Bimini Bay Resort and Bahamas Boat Slips. Enjoy your time with Bahamas Real Estate.

For more information about Bimini Bay Resort and Marina, visit http://bahamasbahamasbahamas.com.
To get idea about Bahamas have a look at Bahamas .
Enjoy Bahamas and Bimini Bay Resort.