ORA-15018, ORA-15072 on 11gR2 grid node 2 when running root.sh

Problem Description
Observation 01:
During installation of Oracle 11gR2 Grid Infrastructure whenever you run root.sh script on second node it fails with error ORA-15018 and ORA-15072 like below.

DiskGroup DATA1 creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 1 regular failure groups, discovered only 0

But root.sh ran successfully on node 1.

Observation 02:
On 2nd node after throwing the error crs fails to start and if you look for logfile rootcrs_2ndnodename.log you will see an entry like,

2010-08-09 15:20:12: Configuring ASM via ASMCA
2010-08-09 15:20:12: Executing as oracle: /u01/app/1120/grid/bin/asmca -silent -diskGroupName DATA1 -diskList ORCL:DATA1 -redundancy EXTERNAL -configureLocalASM
2010-08-09 15:20:12: Running as user oracle: /u01/app/1120/grid/bin/asmca -silent -diskGroupName DATA1 -diskList ORCL:DATA1 -redundancy EXTERNAL -configureLocalASM
2010-08-09 15:20:12: Invoking "/u01/app/1120/grid/bin/asmca -silent -diskGroupName DATA1 -diskList ORCL:DATA1 -redundancy EXTERNAL -configureLocalASM" as user "oracle"
2010-08-09 13:20:16: Configuration of ASM failed, see logs for details

Observation 03:
If you look for /etc/oratab file on second node you will see an interesting observation. There is an entry about +ASM1 instead of +ASM2.

Observation 04:
The following commands on the 2nd node show the ASM disk information correctly.
# /etc/init.d/oracleasm listdisks
# /etc/init.d/oracleasm scandisks
# ls -ltr /dev/oracleasm/disks

Observation 05:
While installing Oracle 11gR2 Grid Infrastructure on ASM, it automatically shows all the disks and disk groups without needing to click on "Disk Discovery Path". It showed disk but it added ORCL in the disk name. Suppose, it was shown disk name as ORCL:DATA1, ORCL:DATA2 etc instead of showing as only DATA1, DATA2.

Cause of the Problem
This problem is actually due to oracle bug. All disks were configured properly but due to bug it could not detect the disk rightly and an ORCL is appended which later caused the problem. There is also possible problem if within /etc/sysconfig/oracleasm file, ORACLEASM_SCANORDER and ORACLEASM_SCANEXCLUDE attribute is configured improperly.

Solution of the Problem
Step 01: Deinstall the oracle clusterware installation.
To know how to deinstall oracle clusterware have a look at http://arjudba.blogspot.com/2010/03/what-to-do-after-failure-of-oracle.html

Step 02: While installation whenever it shows the ASM disk automatically, just click on "Disk discovery path" and type the path manually and let the installer to search path there.

However, if there were improper parameters set within /etc/sysconfig/oracleasm file then do the following:

1. In all RAC nodes modify the /etc/sysconfig/oracleasm with:

ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"

2. In all nodes restart the asmlib by,

# /etc/init.d/oracleasm restart

3. As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force" on all nodes, except the last one. If you have two nodes cluster then only run at first node.
# $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

$su
# $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

4. As root, run "$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode" on last node. This command will zero out OCR and VD disk also. If you have two nodes cluster then run on second node.

# $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode

5. As root, run $GRID_HOME/root.sh on all nodes.
# $GRID_HOME/root.sh
Related Documents
In 11gR2 RAC after server reboot crsd fails to startup on 2nd node
PRKP-1001, CRS-0215 while starting instance using srvctl
NTP and csstd time synchronization option to install Oracle Clusterware 11gR2
cluvfy fails with PRVF-5436 PRVF-9652 Cluster Time Synchronization Services check failed
In 11gR2 Grid root.sh fails with CRS-2674: Start of 'ora.cssd' failed
What to do after failure of Oracle 11gR2 Grid Infrastructure (CRS) Installation
Enable Archive log Mode for RAC database
List of Parameters that must have identical in RAC database
CRS Stack Fails to Start After Reboot ORA-29702 CRS-0184

In 11gR2 RAC after server reboot crsd fails to startup on 2nd node

Problem Description
Oracle Grid Infrastructure, Oracle software Installation went successfully and so creation of oracle database. After installation it is checked CRS daemon on both domains and it is shown all services are online like below.

# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

However, after both server rebooted cluster ready service on second node does not start, but on first node it works fine.

Some output from second node,

# crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

If you look for the Grid alert log on second node it shows,

[crsd(26294)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/dc-db-02/crsd/crsd.log.
2010-08-25 12:06:55.475
[ohasd(25303)]CRS-2765:Resource 'ora.crsd' has failed on server 'dc-db-02'.
2010-08-25 12:06:56.553
[crsd(26305)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/dc-db-02/crsd/crsd.log.
2010-08-25 12:06:57.499
[ohasd(25303)]CRS-2765:Resource 'ora.crsd' has failed on server 'dc-db-02'.
2010-08-25 12:06:57.499
[ohasd(25303)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

If you look for crsd.log file, it will show

2010-08-25 12:06:56.552: [  OCRASM][1855820272]proprasmo: Error in open/create file in dg [DATA]
[  OCRASM][1855820272]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup

2010-08-25 12:06:56.553: [  OCRASM][1855820272]proprasmo: kgfoCheckMount returned [7]
2010-08-25 12:06:56.553: [  OCRASM][1855820272]proprasmo: The ASM instance is down
2010-08-25 12:06:56.553: [  OCRRAW][1855820272]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2010-08-25 12:06:56.553: [  OCRRAW][1855820272]proprioo: No OCR/OLR devices are usable
2010-08-25 12:06:56.554: [  OCRASM][1855820272]proprasmcl: asmhandle is NULL
2010-08-25 12:06:56.554: [  OCRRAW][1855820272]proprinit: Could not open raw device 
2010-08-25 12:06:56.554: [  OCRASM][1855820272]proprasmcl: asmhandle is NULL
2010-08-25 12:06:56.554: [  OCRAPI][1855820272]a_init:16!: Backend init unsuccessful : [26]
2010-08-25 12:06:56.554: [  CRSOCR][1855820272] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
] [7]
2010-08-25 12:06:56.554: [    CRSD][1855820272][PANIC] CRSD exiting: Could not init OCR, code: 26
2010-08-25 12:06:56.554: [    CRSD][1855820272] Done.

Cause of the Problem
Whenever you look for "CRS-4535: Cannot communicate with Cluster Ready Services" error immediate investigate in the grid alert log as well as crsd.log file. There you would get more information regarding CRS-4535 error. From the crsd.log file we see the oracle error message
ORA-15077: could not locate ASM instance serving a required diskgroup

If we try to run ls command after inside amdcmd it fails with ASMCMD-08102

$ export ORACLE_SID=+ASM2
$ amdcmd
ASMCMD> ls
ASMCMD-08102: no connection to ASM; command requires ASM to run"

Whenever it is tried to start ASM instance manually on the second node you get error message like below.

amdcmd> startup;
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpsemsper

This error message may mislead you because there is sufficient space in your disk. df -h command proves that. This error entirely related to kernel parameter settings. If you check your semaphores setting you see there does not have enough semaphores allowed for creating more processes.

Solution of the problem
As this problem is lower settings of semaphores value so solution is to increase the semaphores value.

- Check your current semapores settings by looking for value of the parameter "kernel.sem" inside the file /etc/sysctl.conf
# cat /etc/sysctl.conf

or issue,
# /sbin/sysctl -a | grep sem

- Modify SEMMNI value in the /etc/sysctl.conf like below.

# vi /etc/sysctl.conf
kernel.sem = 250 32000 100 200

On Red Hat Linux system, in order the affect the setting of the value immediately use,
# /sbin/sysctl -p

Now starting up the asm instance is just fine and also CRS daemon is ok. Restart the node to make sure everything is working perfect.
Related Documents
ORA-15018, ORA-15072 on 11gR2 grid node 2 when running root.sh
PRKP-1001, CRS-0215 while starting instance using srvctl
NTP and csstd time synchronization option to install Oracle Clusterware 11gR2
cluvfy fails with PRVF-5436 PRVF-9652 Cluster Time Synchronization Services check failed
In 11gR2 Grid root.sh fails with CRS-2674: Start of 'ora.cssd' failed
What to do after failure of Oracle 11gR2 Grid Infrastructure (CRS) Installation
Enable Archive log Mode for RAC database
List of Parameters that must have identical in RAC database
CRS Stack Fails to Start After Reboot ORA-29702 CRS-0184