问题描述
在HP-UX上执行asmcmd会出现下面的错误，为了解决这个bug，需要应用补丁20948770，内容如下：
ohsdb1#[/home/orgrid]asmcmdUse of uninitialized value in subroutine entry at /pgold/orgrid/oracle/product/121/lib/asmcmdbase.pm line 5086.
panic: no address for 'ASMCMDGetConnDetails' in 'DynaLoader' at /pgold/orgrid/oracle/product/121/lib/asmcmdbase.pm line 5086.
ohsdb1#[/home/orgrid]
打完补丁后，在启动crs时出现下面的错误，d.bin后台进程也没有
ohsdb1#[/]/pgold/orgrid/oracle/product/121/bin/crsctl start crs
CRS-6706: Oracle Clusterware Release patch level ('333253490') does not match Software patch level ('3700167791'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
ohsdb1#[/]
问题分析
发现补丁20948770已经应用完毕，也可以查到。经了解，是通过opatch一个节点一个节点正常去打的，应用补丁过程中也没出现错误，出现这样的问题也甚是奇怪。
ohsdb1#[/home/orgrid]ohsdb1#[/home/orgrid]kfod op=patchlvl
-------------------
Current Patch level
===================
333253490
ohsdb1#[/home/orgrid]
ohsdb1#[/home/orgrid]kfod op=patches
---------------
List of Patches
===============
19769480
20299023
20831110
20948770
21359755
21948354
22291127
23054246
24006101
24732082
24828633
ohsdb1#
crsctl query crs softwarepatch
ohsdb1#[/]/pgold/orgrid/oracle/product/121/bin/crsctl query crs softwarepatch
Oracle Clusterware patch level on node xxywa1 is [3700167791].
crsctl query crs releasepatc
Oracle Clusterware release patch level is [333253490] and the complete list of patches [19769480 20299023 20831110 20948770 21359755 21948354 22291127 23054246 24006101 24732082 24828633 ] have been applied on the local node.
ohsdb1#[/]/pgold/orgrid/oracle/product/121/bin/crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.1.0.2.0]
ohsdb1#[/]
解决问题
1.)通过opatch回滚20948770，回滚后，crs softwarepath和crs releasepatch是一致的
ohsdb1#[/pgold/patches/20948770]opatch rollback -id 20948770ohsdb1#[/home/orgrid]crsctl query crs softwarepatch
Oracle Clusterware patch level on node xxywa1 is [3700167791].
ohsdb1#[/home/orgrid] crsctl query crs releasepatch
Oracle Clusterware release patch level is [3700167791] and the complete list of patches [19769480 20299023 20831110 21359755 21948354 22291127 23054246 24006101 24732082 24828633 ] have been applied on the local node.
ohsdb1#[/home/orgrid]crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.1.0.2.0]
ohsdb1#[/home/orgrid]
2.)使用opatchauto分析和应用补丁20948770
ohsdb1#[/pgold/patches/20948770]/pgold/orgrid/oracle/product/121/OPatch/opatchauto apply -analyze
OPatchauto session is initiated at Wed Apr 12 23:07:35 2017
System initialization log file is /pgold/orgrid/oracle/product/121/cfgtoollogs/opatchautodb/systemconfig2017-04-12_11-07-43PM.log.
Session log file is /pgold/orgrid/oracle/product/121/cfgtoollogs/opatchauto/opatchauto2017-04-12_11-08-16PM.log
The id for this session is FIYC
Executing OPatch prereq operations to verify patch applicability on home /pgold/orgrid/oracle/product/121
Patch applicablity verified successfully on home /pgold/orgrid/oracle/product/121
Verifying patch inventory on home /pgold/orgrid/oracle/product/121
No step execution required.........
Patch inventory verified successfully on home /pgold/orgrid/oracle/product/121
OPatchAuto successful.
--------------------------------Summary--------------------------------
Analysis for applying patches has completed successfully:
Host:xxywa1
CRS Home:/pgold/orgrid/oracle/product/121
==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /pgold/patches/20948770
Log: /pgold/orgrid/oracle/product/121/cfgtoollogs/opatchauto/core/opatch/opatch2017-04-12_23-08-28PM_1.log
OPatchauto session completed at Wed Apr 12 23:08:44 2017
Time taken to complete the session 1 minute, 12 seconds
ohsdb1#[/pgold/patches/20948770]/pgold/orgrid/oracle/product/121/OPatch/opatchauto apply
OPatchauto session is initiated at Wed Apr 12 23:09:23 2017
System initialization log file is /pgold/orgrid/oracle/product/121/cfgtoollogs/opatchautodb/systemconfig2017-04-12_11-09-30PM.log.
Session log file is /pgold/orgrid/oracle/product/121/cfgtoollogs/opatchauto/opatchauto2017-04-12_11-10-02PM.log
The id for this session is YXD3
Executing OPatch prereq operations to verify patch applicability on home /pgold/orgrid/oracle/product/121
Patch applicablity verified successfully on home /pgold/orgrid/oracle/product/121
Verifying patch inventory on home /pgold/orgrid/oracle/product/121
No step execution required.........
Patch inventory verified successfully on home /pgold/orgrid/oracle/product/121
Bringing down CRS service on home /pgold/orgrid/oracle/product/121
Prepatch operation log file location: /pgold/orgrid/oracle/product/121/cfgtoollogs/crsconfig/crspatch_xxywa1_2017-04-12_11-10-29PM.log
CRS service brought down successfully on home /pgold/orgrid/oracle/product/121
Start applying binary patch on home /pgold/orgrid/oracle/product/121
Binary patch applied successfully on home /pgold/orgrid/oracle/product/121
Starting CRS service on home /pgold/orgrid/oracle/product/121
Postpatch operation log file location: /pgold/orgrid/oracle/product/121/cfgtoollogs/crsconfig/crspatch_xxywa1_2017-04-12_11-16-15PM.log
CRS service started successfully on home /pgold/orgrid/oracle/product/121
Verifying patches applied on home /pgold/orgrid/oracle/product/121
No step execution required.........
Patch verification completed with warning on home /pgold/orgrid/oracle/product/121
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:xxywa1
CRS Home:/pgold/orgrid/oracle/product/121
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /pgold/patches/20948770
Log: /pgold/orgrid/oracle/product/121/cfgtoollogs/opatchauto/core/opatch/opatch2017-04-12_23-11-54PM_1.log
OPatchauto session completed at Wed Apr 12 23:19:41 2017
Time taken to complete the session 10 minutes, 19 seconds
ohsdb1#[/pgold/patches/20948770]
3.)查看softwarepatch和releasepatch
ohsdb1#[/home/orgrid]crsctl query crs softwarepatch
Oracle Clusterware patch level on node xxywa1 is [333253490].
ohsdb1#[/home/orgrid]crsctl query crs releasepatch
Oracle Clusterware release patch level is [333253490] and the complete list of patches [19769480 20299023 20831110 20948770 21359755 21948354 22291127 23054246 24006101 24732082 24828633 ] have been applied on the local node.
ohsdb1#[/home/orgrid]
补丁应用后，asmcmd正常，至此问题完美解决
ohsdb1#[/home/orgrid]asmcmd
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 1024 4096 1048576 30720 29893 10240 9826 0 Y SYSTEMDG/ASMCMD>
ASMCMD> showclustermode
ASM cluster : Flex mode enabled
ASMCMD>
总结和建议
12c以后建议通过opatchauto去应用补丁，这个很省事，记得先更应用前记得要更新opatch版本。
ASMCMD Command Is Failing On 12.1.0.2 ASM Flex Architecture (HP-UX Itanium) (Doc ID 2016132.1)
