Oracle ASM Corruption: How To Fix The ASM Disk HEADER_STATUS From FORMER To MEMBER.

ASM Corruption: Case #1: How To Fix The ASM Disk HEADER_STATUS From FORMER or PROVISIONED To MEMBER.

Author: Esteban D. Bernal

This document explains, in detail, the steps required (with an example) to fix the ASM disk header from FORMER to MEMBER under the next scenarios:

A)Diskgroup was dropped by accident using the “SQL> drop <DG name> diskgroup;” statement.

B) Or under strange situations as described in the next bug

Bug.13331814ASM DISKS TURNED INTO FORMER WHILE DISKGROUP IS MOUNTED.

Example

1) You created a diskgroup with the next disk member

[grid@asmlnx1 ~]$ sqlplus "/as sysasm"

SQL*Plus: Release 12.1.0.1.0 Production on Sun May 18 16:16:57 2014

Connected to

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production With the Automatic Storage Management option

SQL> set linesize 100
SQL> column path format a50
SQL> select name, path , header_status from v$asm_disk where path like 'ORCL:SAN_A_DISK_1';

NAMEPATHHEADER_STATU

ORCL: SAN_A_DISK_1PROVISIONED

SQL> create diskgroup TESTDG external redundancy disk 'ORCL:SAN_A_DISK_1';

Diskgroup created.

SQL> select group_number, name, state from v$asm_diskgroup where name like 'TESTDG';

GROUP_NUMBER NAMESTATE

1 TESTDGMOUNTED

SQL> set linesize 100
SQL> column path format a50
SQL> select name, path , header_status from v$asm_disk where group_number = 1;

NAMEPATHHEADER_STATU

SAN_A_DISK_1ORCL: SAN_A_DISK_1MEMBER

2) Then later, the diskgroup is dropped by accident and cannot be mounted again

SQL> drop diskgroup TESTDG;

Diskgroup dropped.

SQL> alter diskgroup TESTDG mount; alter diskgroup TESTDG mount

ERROR at line 1

ORA-15032: not all alterations performed
ORA-15017: diskgroup "TESTDG" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "TESTDG"

3) The original disk member is now shown as a FORMER disk

SQL> set linesize 100
SQL> column path format a50
SQL> select name, path , header_status from v$asm_disk where path like 'ORCL:SAN_A_DISK_1';

4) Kfed also reports the FORMER status

[grid@asmlnx1 ~]$ kfed read /dev/oracleasm/disks/SAN_A_DISK_1 | head -25

Kfbh.block.obj:2147483648; 0x008: disk=0

Kfbh.check:2862642817; 0x00c: 0xaaa07681

Kfdhdb.driver.provstr: ORCLDISKSAN_A_DISK_1; 0x000: length=20

Kfdhdb.driver.reserved[0]: 1598964051; 0x008: 0x5f4e4153

Kfdhdb.driver.reserved[1]: 1229217601; 0x00c: 0x49445f41

Kfdhdb.driver.reserved[2]:828328787; 0x010: 0x315f4b53 kfdhdb.driver.reserved[3]:0; 0x014: 0x00000000 kfdhdb.driver.reserved[4]:0; 0x018: 0x00000000 kfdhdb.driver.reserved[5]:0; 0x01c: 0x00000000 kfdhdb.compat:168820736; 0x020: 0x0a100000

Kfdhdb.dskname:SAN_A_DISK_1; 0x028: length=12

Kfdhdb.grpname:TESTDG; 0x048: length=6 kfdhdb.fgname:SAN_A_DISK_1; 0x068: length=12

5)The ASM disk header needs to be patched/updated with the HEADER_STATUS = “MEMBER”In order to mount the diskgroup again as follow:

5.1) Backup the first 10 MB of the associated disks using the dd command

[grid@asmlnx1 ~]$ mkdir /tmp/asm

[grid@asmlnx1~]$

of=/tmp/SAN_A_DISK_1.dump bs=1048576 count=50 50+0 records in

50+0 records out

52428800 bytes (52 MB) copied, 2.0715 seconds, 25.3 MB/s 5.2) Generate the template patch from the FORMER disk:

/tmp/asm/patch_SAN_A_DISK_1.txt

[grid@asmlnx1~]$ ls -l /tmp/asm/patch_SAN_A_DISK_1.txt

-rw-r--r-- 1 grid oinstall 6594 Apr 10 16:48 /tmp/asm/patch_SAN_A_DISK_1.txt

5.3)Edit the file (/tmp/asm/patch_SAN_A_DISK_1.txt) and change the code # from 4 to 3 and the status from FORMER to MEMBER in the next line (make sure you save the changes):

Original line

New line

Note: #4 stands for “FORMER” & #3 stands for “MEMBER”

5.4) Verify the new values in the patch file

[grid@asmlnx1~]$ grep kfdhdb.hdrsts /tmp/asm/patch_SAN_A_DISK_1.txt

5.5) Now you can patch the FORMER disk using the “/tmp/asm/patch_SAN_A_DISK_1.txt” patch file

text=/tmp/asm/patch_SAN_A_DISK_1.txt

5.6)Verify the new values in the real disk (ASM disk header was fixed with the HEADER_STATUS= “MEMBER”):

[grid@asmlnx1~]$KfedRead/dev/oracleasm/disks/SAN_A_DISK_1|Grep kfdhdb.hdrsts

6) Now you can mount the diskgroup

[grid@asmlnx1 ~]$ sqlplus "/as sysasm"

SQL*Plus: Release 12.1.0.1.0 Production on Sun May 18 16:31:25 2014

Connected to

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production With the Automatic Storage Management option

SQL> alter diskgroup TESTDG mount;

Diskgroup altered.

SQL> set linesize 100
SQL> column path format a50
SQL> select name, path , header_status from v$asm_disk where path like 'ORCL:SAN_A_DISK_1';
SQL> alter diskgroup TESTDG mount;

Diskgroup altered.

Annex: Syntax to create the ASMLIB disk

[root@asmlnx1 ~]# /etc/init.d/oracleasm querydisk -p SAN_A_DISK_1 Disk "SAN_A_DISK_1" is a valid ASM disk

/dev/asmdisk1_4k_udev_p1: LABEL="SAN_A_DISK_1" TYPE="oracleasm"

[root@asmlnx1 ~]# /etc/init.d/oracleasm createdisk SAN_A_DISK_1

/dev/asmdisk1_4k_udev_p1

Marking disk "SAN_A_DISK_1" as an ASM disk:[ OK ]

DBRECOVER Recovery Options

For Oracle incidents, start with the DBRECOVER for Oracle trial to verify table visibility, row previews, and export readiness on copied datafiles. For MySQL and InnoDB incidents, DBRECOVER for MySQL is free software and can inspect.ibd files, ibdata1, and database directories locally.

When the case is urgent, preserve the original files first, work from copies, and contact paid emergency support with the database version, platform, error messages, file list, and recovery objective.