APPLIES TO
Oracle Database - Enterprise Edition
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Information in this document applies to any platform.
PURPOSE
Self-debugging steps when a diskgroup cannot be mounted due to error ORA-15063
ORA-15063: ASM discovered an insufficient number of disks for diskgroup s%
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "%" is missingTROUBLESHOOTING STEPS
SECTION A - Getting started
Start by refering NOTE 452770.1 "TROUBLESHOOTING - ASM disk not found/visible/discovered issues "
Firstly identify all disks being part of the affected diskgroup by looking at last successful mount in alert_+ASM*.log.
You should search for a section as below
SQL> ALTER DISKGROUP <DGNAME1> MOUNT /* asm agent *//* {0:0:214} */NOTE: cache registered group DATA number=1 incarn=0x44bef6bb
NOTE: cache began mount (not first) of group DATA number=1 incarn=0x44bef6bb
NOTE: Loaded library: /opt/Oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Assigning number (1,0) to disk (ORCL: DATA01P)
NOTE: Assigning number (1,1) to disk (ORCL: DATA02P)
NOTE: Assigning number (1,2) to disk (ORCL: DATA03P)
NOTE: Assigning number (1,3) to disk (ORCL: DATA04P)
NOTE: Assigning number (1,4) to disk (ORCL: DATA05P)
..
NOTE: cache opening disk 0 of grp 1: DATA01P label: DATA01P
NOTE: cache opening disk 1 of grp 1: DATA02P label: DATA02P
..
SUCCESS: DISKGROUP <DGNAME1> was mounted
NOTE: When ASMLIB is not used the path to ASM disk is specified within the mount section
NOTE: cache opening disk 1 of grp 1: REDO3_0001 path:/dev/mpath/3600601600ba12c00d4b784363e69e211
NOTE: cache opening disk 2 of grp 1: REDO3_0002 path:/dev/mpath/3600601600ba12c00d4b784363e69e212
...
Isolate the device(s) reported as "missing" as note 452770.1 suggested.
Finally start your checks as follow
A1) If there is any IO/storage/multipathing errors reported in OS logs - investigate and fix them.
This step is mandatory as usually ORA-15063/ORA-15042 are caused by underlying IO/storage errors .
A2) If devices used by ASM disks are properly presented and configured at OS level.
If additionally "ORA-15075: disk(s) are not visible cluster-wide" is reported, make sure that all devices are cluster-wide visible.
A3) If all ASM disks have appropriate permissions (eg: they should be owned by grid owner)
If ownership of ASM disk(s) has been changed for whatever reason, please correct that.
A4) If/how the "missing" device(s) is reported when querying v$asm_disks
-----------------------------------------------------------------------------------
If the device(s) is reported with status
=> "PROVISIONED/CANDIDATE" - this means the header of ASM disk(s) is damaged.
-> investigate the IO problems behind the corruption - see step A1. Oracle never wipes out its metadata!! A checksum is made for every write before being accepted.
-> check the header status, in order to confirm the damage
$> kfed read <path_to_your_missing_devices>Kfbh.endian: 0; 0x000: 0x00
Kfbh.hard: 0; 0x001: 0x00
Kfbh.type: 0; 0x002: KFBTYP_INVALID
Kfbh.datfmt: 0; 0x003: 0x00
Kfbh.block.blk: 0; 0x004: blk=0
Kfbh.block.obj: 0; 0x008: file=0
....
-> try to repair the header and see if diskgroup can be mounted
$> kfed repair <path_to_your_missing_devices>-> check the if there is additional corruptions reported by ASM (eg ORA-15196) or by your database - as IO/storage problems could affect more than one block.
If any corruption is seen please open a SR to Oracle Support.
NOTE
1) When non-default AU size is used AUSZ=<au_size> must be specified with each KFED command.
2) "kfed repair" works for 11g ONLY!
=> "UNKNOWN/IGNORED" - this means the ASM disk(s) is not seen at OS level.
-> review steps A1, A2 and A3
-----------------------------------------------------------------------------------
A5) If asm_diskstring is still properly set.
On Windows configuration, you can also refer NOTE 880061.1 "ASM Is Unable To Detect SCSI Disks On Windows"
SECTION B - ASMLIB is used
When ASMLIB is used, follow the above steps (section A) and also check the errors associated with ORA-15063:
B1) ORA-15183 Unable to initialize the ASMLIB in Oracle/ORA-15183: ASMLIB initialization error [driver/agent not installed]
Refer: NOTE 340519.1 Cannot Start ASM Ora-15063/ORA-15183
B2) ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Check your ASMLIB health.
=> correctness of installed rpm's
=> correctness of symlinks - all nodes should show
# ls -l /etc/sysconfig/oracleasmLrwxrwxrwx 1 root root 24 Sep 18 22:10 /etc/sysconfig/oracleasm -> oracleasm-_dev_oracleas
=> correctness of ASMLIB configuration (/etc/sysconfig/oracleasm) - when multipathing is used
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="dm"
# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"B3) Check if ASMLIB disks are listed under /dev/oracleasm/disks
=> devices under /dev/oracleasm/disks/* must be reported as dm devices on all nodes (not single path device -sd*-).If not, please correct that! (see step B2)
$> ls -al /dev/oracleasm/disksBrw-rw---- 1 grid dba 253, 29 Feb 12 11:44 /dev/oracleasm/disks/DATA01P
Brw-rw---- 1 grid dba 253, 35 Feb 12 11:44 /dev/oracleasm/disks/DATA02P
Brw-rw---- 1 grid dba 253, 27 Feb 15 16:04 /dev/oracleasm/disks/DATA03P
Brw-rw---- 1 grid dba 253, 24 Feb 12 11:44 /dev/oracleasm/disks/DATA04P
Brw-rw---- 1 grid dba 253, 25 Feb 12 11:44 /dev/oracleasm/disks/DATA05P
=> If one of your ASMLIB disk(s) is missing from the above output, first try to re-scan devices, as root:
# /etc/init.d/oracleasm scandisks=> If ASMLIB disk(s) is still missing from /dev/oracleasm/disks, engage your sysadmin to investigate this (see steps A1, A2, A3).
B4) Check if ASMLIB disk(s) has the correct ASMLIB stamp and status
$> kfed read <ASMLIB_device> |grep provstrKfdhdb.driver.provstr: ORCLDISK<diskname>; 0x000: length=20
$> kfed read <ASMLIB_device> | egrep 'kfbh.type|kfdhdb.dskname|kfdhdb.hdrsts'Kfbh.type: 1; 0x002: KFBTYP_DISKHEAD
Kfdhdb.dskname: DATA01P; 0x028: length=14
Kfdhdb.hdrsts: 3; 0x027: KFDHDR_MEMBER
=> If the output is "kfdhdb.driver.provstr: ORCLCLRD" (but kfdhdb.hdrsts= MEMBER and kfbh.type=KFBTYP_DISKHEAD) then your disk was deleted using "oracleasm deletedisk".
=> If kfbh.type = KFBTYP_INVALID -> see step A4) and check if "kfed repair" could fix the problem.
B5)Refer also the below documents
NOTE: 398622.1 ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
NOTE: 1384504.1 Mount ASM Disk Group Fails: ORA-15186, ORA-15025, ORA-15063
NOTE: 967461.1 "Multipath: error getting device" seen in OS log causes ASM/ASMlib to shutdown by itself
NOTE: 1526920.1 ORA-15186 ORA-15063 on node 2
SECTION C - Additional notes to review
If the above checks are done, but error still persists, please review also the below notes, depending on your configuration/situation:
NOTE: 577526.1 ORA-15063 ASM Discovered An Insufficient Number Of Disks For Diskgroup using NetApp Storage
NOTE: 784776.1 ORA-15063 When Mounting a Diskgroup After Storage Cloning ( BCV / Split Mirror / SRDF / HDS / Flash Copy )
NOTE: 555918.1 ORA-15038 On Diskgroup Mount After Node Eviction
NOTE: 1484723.1 ASM Candidate Raw Device Is Not Presented As A RAC Cluster Wide Shared character Devices On Unix.
NOTE: 1534211.1 ORA-15017 and ORA-15063 errors for unused diskgroups in 11.2
NOTE: 1487443.1 Mounting Diskgroup Fails With ORA-15063 and V$ASM_DISK Shows PROVISIONED
NOTE: 742832.1 AIX: After changing Multipathing drivers from RDAC to MPIO ASM discovered an insufficient number of disks
NOTE: 1276913.1 Unable to discover or use raw devices for ASM in HP-UX Itanium in 11.2.0.2 ( ORA-15063 )
SECTION D - Information to be collected when are you going to open a SR
If you are not able to fix the problem on your own, please collect the below information and raise a SR to Oracle Support
D1) alert_+ASM*.log (from all nodes if RAC)
D2) script#1 from NOTE 470211.1 How To Gather/Backup ASM Metadata In A Formatted Manner version 10.1, 10.2, 11.1 & 11.2?
D3) KFED reports
#! /bin/shRm /tmp/kfed_DH.out /tmp/kfed_BK.out
For I in `ls <your_path_to_asm_disks>`
Do
Echo $I >> /tmp/kfed_DH.out
Kfed read $I >> /tmp/kfed_DH.out
Echo $I >> /tmp/kfed_BK.out
Kfed read $I aun=1 blkn=254 >> /tmp/kfed_BK.out
Done
Run kfed.sh in as GRID/ASM owner. Upload /tmp/kfed_DH.out, /tmp/kfed_BK.out
! Pay attention to non-default AU size - if a non-default AU size is used the you must specify it. (see note 1485597.1 "ASM tools used by Support: KFOD, KFED, AMDU")
D4) ASMLIB information
NOTE: 869526.1 Collecting The Required Information For Support To Troubleshot ASM/ASMLIB Issues.
D5) List of your ASM devices
$> ls -al <path_to_ASM_devices>D6) OS logs (from all nodes if this is RAC configuration)
SECTION E - Disk is reported as MISSING after a failed disk addition
If you are facing ORA-15063 after a failed disk addition, please collect the below information and raise a SR to Oracle Support
E1) alert_+ASM*.log (from all nodes if RAC)
E2) script#1 from NOTE 470211.1 How To Gather/Backup ASM Metadata In A Formatted Manner version 10.1, 10.2, 11.1 & 11.2?
E3) KFED reports
#! /bin/shRm /tmp/kfed_*.out
For I in `ls <your_path_to_asm_disks>`
Do
Echo $I >> /tmp/kfed_DH.out
Kfed read $I >> /tmp/kfed_DH.out
Echo $I >> /tmp/kfed_BK.out
Kfed read $I aun=1 blkn=254 >> /tmp/kfed_BK.out
Echo $I >> /tmp/kfed_PST.out
Kfed read $I aun=1 blkn=2 >> /tmp/kfed_PST.out
Echo $I >> /tmp/kfed_FS.out
Kfed read $I blkn=1 >> /tmp/kfed_FS.out
Echo $I >> /tmp/kfed_FD.out
Kfed read $I aun=2 blkn=1 >> /tmp/kfed_FD.out
Echo $I >> /tmp/kfed_DD.out
Kfed read $I aun=2 blkn=0 >> /tmp/kfed_DD.out ##there might be more than one block needed if a large number of disks -> this might be asked later by Oracle Support
Done
Run kfed.sh in as GRID/ASM owner. Upload /tmp/kfed_*.out
! Pay attention to non-default AU size - if a non-default AU size is used the you must specify it. (see note 1485597.1 "ASM tools used by Support: KFOD, KFED, AMDU")
E4) AMDU output
Amdu -diskstring '<ASM_DISKSTRING>' -dump '<DISKGROUP_NAME>' -noimage
Amdu -diskstring '<ASM_DISKSTRING>' -print <DISKGROUP_NAME>.F2.V0.C2 > DG.amdu
####F2.V0.C2 --> This will only extract up to 16 disks information. If there is a large number of disks, a larger output is neededDBRECOVER Recovery Options
For Oracle incidents, start with the DBRECOVER for Oracle trial to verify table visibility, row previews, and export readiness on copied datafiles. For MySQL and InnoDB incidents, DBRECOVER for MySQL is free software and can inspect.ibd files, ibdata1, and database directories locally.
When the case is urgent, preserve the original files first, work from copies, and contact paid emergency support with the database version, platform, error messages, file list, and recovery objective.