Clusterware startup process:

Clusterware Stratup :

- When a node of an Oracle Clusterware cluster restarts, OHASD is started by platform-specific means. OHASD accesses OLR (Oracle Local Registry) stored on the local file system to get the data needed to complete OHASD initialization

-  OHASD brings up GPNPD and CSSD. CSSD accesses the GPNP Profile stored on the local file system which contains the following vital bootstrap data;
a.  ASM_DISKSTRING parameter (if specified) to locate the disks on which ASM disks are configured.
b.  ASM SPFILE location : Name of the diskgroup containing ASM spfile

c.  Voting Disk Files location : ASM

– CSSD scans the headers of all ASM disks ( as indicated in ASM_DISKSTRING in GPnP profile) to identify the disk containing the voting file.  Using the pointers in ASM disk headers, the Voting Files locations on ASM Disks are accessed by CSSD and CSSD is able to complete initialization and start or join an existing cluster.

–To read the ASM spfile during the ASM instance startup, it is not necessary to open the disk group. All information necessary to access the data is stored in the device’s header. OHASD reads the header of the ASM disk containing ASM SPfile (as read from GPnP profile) and using the pointers in disk header, contents of ASM spfile are read. Thereafter, ASM instance is started.

–  With an ASM instance operating and its Diskgroups mounted, access to Clusterware’s OCR is available to CRSD.

–  OHASD starts CRSD with access to the OCR in an ASM Diskgroup.

–  Clusterware completes initialization and brings up other services under its control.

Demonstration :

In my environment, the ASM disk group DATA created with EXTERNAL  redundancy is used exclusively for ASM spfile, voting and OCR files:

- Let us read  gpnp profile to find out the location of ASM SPfile

[grid@host01 peer]$ cd /u01/app/11.2.0/grid/gpnp/host01/profiles/peer

                                  gpnptool getpval -asm_spf

Warning: some command line parameters were defaulted. Resulting command line:

         /u01/app/11.2.0/grid/bin/gpnptool.bin getpval -asm_spf -p=profile.xml -o-

+DATA/cluster01/asmparameterfile/registry.253.793721441

– Let us find out the disks in DATA diskgroup

[grid@host01 peer]$ asmcmd lsdsk -G DATA

Path

ORCL:ASMDISK01

ORCL:ASMDISK010

ORCL:ASMDISK02

ORCL:ASMDISK03

ORCL:ASMDISK04

ORCL:ASMDISK09

– Let us find out which ASM disk maps to which partition

   Note down major/minor device numbers of the disks in DATA diskgroup

[root@host01 ~]# ls -lr /dev/oracleasm/disks/*

brw-rw—- 1 grid asmadmin 8, 26 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK09

brw-rw—- 1 grid asmadmin 8, 25 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK08

brw-rw—- 1 grid asmadmin 8, 24 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK07

brw-rw—- 1 grid asmadmin 8, 23 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK06

brw-rw—- 1 grid asmadmin 8, 22 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK05

brw-rw—- 1 grid asmadmin 8, 21 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK04

brw-rw—- 1 grid asmadmin 8, 19 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK03

brw-rw—- 1 grid asmadmin 8, 18 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK02

brw-rw—- 1 grid asmadmin 8, 31 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK014

brw-rw—- 1 grid asmadmin 8, 30 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK013

brw-rw—- 1 grid asmadmin 8, 29 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK012

brw-rw—- 1 grid asmadmin 8, 28 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK011

brw-rw—- 1 grid asmadmin 8, 27 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK010

brw-rw—- 1 grid asmadmin 8, 17 Nov  8 09:35 /dev/oracleasm/disks/ASMDISK01

- Let us find out the major/minor device numbers of various disk partitions

[root@host01 ~]# ls -lr /dev/sdb*

brw-r—– 1 root disk 8, 25 Nov  8 09:35 /dev/sdb9

brw-r—– 1 root disk 8, 24 Nov  8 09:35 /dev/sdb8

brw-r—– 1 root disk 8, 23 Nov  8 09:35 /dev/sdb7

brw-r—– 1 root disk 8, 22 Nov  8 09:35 /dev/sdb6

brw-r—– 1 root disk 8, 21 Nov  8 09:35 /dev/sdb5

brw-r—– 1 root disk 8, 20 Nov  8 09:35 /dev/sdb4

brw-r—– 1 root disk 8, 19 Nov  8 09:35 /dev/sdb3

brw-r—– 1 root disk 8, 18 Nov  8 09:35 /dev/sdb2

brw-r—– 1 root disk 8, 31 Nov  8 09:35 /dev/sdb15

brw-r—– 1 root disk 8, 30 Nov  8 09:35 /dev/sdb14

brw-r—– 1 root disk 8, 29 Nov  8 09:35 /dev/sdb13

brw-r—– 1 root disk 8, 28 Nov  8 09:35 /dev/sdb12

brw-r—– 1 root disk 8, 27 Nov  8 09:35 /dev/sdb11

brw-r—– 1 root disk 8, 26 Nov  8 09:35 /dev/sdb10

brw-r—– 1 root disk 8, 17 Nov  8 09:35 /dev/sdb1

brw-r—– 1 root disk 8, 16 Nov  8 09:35 /dev/sdb

– Now we can find out the partitions mapping to various ASM disks by matching their

   major/minor device numbers

 ASMDISK01    8,17     /dev/sdb1

 ASMDISK02    8,18     /dev/sdb2

 ASMDISK03    8,19     /dev/sdb3

 ASMDISK04    8,21     /dev/sdb5

 ASMDISK09    8,26     /dev/sdb10

 ASMDISK10    8,27     /dev/sdb11

– Let’s scan the headers of those devices:

[root@host01 ~]#  kfed read /dev/sdb1 | grep -E ‘spf|ausize’

kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000

kfdhdb.spfile:                        0 ; 0x0f4: 0x00000000

kfdhdb.spfflg:                        0 ; 0x0f8: 0x00000000

[root@host01 ~]#  kfed read /dev/sdb2 | grep -E ‘spf|ausize’

kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000

kfdhdb.spfile:                        0 ; 0x0f4: 0x00000000

kfdhdb.spfflg:                        0 ; 0x0f8: 0x00000000

[root@host01 ~]#  kfed read /dev/sdb3 | grep -E ‘spf|ausize’

kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000

kfdhdb.spfile:                       16 ; 0x0f4: 0x00000010

kfdhdb.spfflg:                        1 ; 0x0f8: 0x00000001

In the output above, we see that

     the device /dev/sdb3 contains a copy of the ASM spfile (spfflg=1).

     The ASM spfile location starts at the disk offset of 16 (spfile=16)

Considering the allocation unit size (kfdhdb.ausize = 1M), let’s dump the ASM spfile from the device:

[root@host01 ~]#  dd if=/dev/sdb3 of=spfileASM_Copy2.ora skip=16  bs=1M count=1

1+0 records in

1+0 records out

1048576 bytes (1.0 MB) copied, 0.170611 seconds, 6.1 MB/s

[root@host01 ~]# strings spfileASM_Copy2.ora

+ASM1.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value

+ASM2.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value

+ASM3.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value

+ASM3.asm_diskgroups=’FRA’#Manual Mount

+ASM2.asm_diskgroups=’FRA’#Manual Mount

+ASM1.asm_diskgroups=’FRA’#Manual Mount

*.asm_power_limit=1

*.diagnostic_dest=’/u01/app/grid’

*.instance_type=’asm’

*.large_pool_size=12M

*.remote_login_passwordfile=’EXCLUSIVE’

a:O/

 The same technique is used to access the Clusterware voting files which are also stored in an ASM disk group. In this case, Clusterware does not need a running ASM instance to access the cluster voting files:

Let’s check the location of voting disk :

[grid@host01 peer]$ crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

–  —–    —————–                ——— ———

 1. ONLINE   243ec3b2a3cf4fbbbfed6f20a1ef4319 (ORCL:ASMDISK01) [DATA]

Located 1 voting disk(s).

– Since above query shows that voting disk is stored on ASMDISK01 which maps to /dev/sdb1,

   we will scan the header of /dev/sdb1

[root@host01 ~]#  kfed read /dev/sdb1 | grep vf

kfdhdb.vfstart:                      96 ; 0x0ec: 0x00000060

kfdhdb.vfend:                       128 ; 0x0f0: 0x00000080

Here we can see that voting disk resides on /dev/sdb1 .

Once the voting disk is accessible and ASM is started using the SPfile read above, rest of the resources on the node can be started after reading the Oracle Local Registry (OLR) on the node.