RAC中磁盘绑定与权限管理

A.Linux下的mutipath也是我们常用的多路径软件，可以通过rpm或yum安装
  yum install kpartx  -y
  yum install device-mapper-multipath -y
  chkconfig multipathd on
  service multipathd start

配置文件为/etc/multipath.conf，如果这个文件没有，可以通过mpathconf --enable命令生成。文件/etc/multipath/bindings和/etc/multipath/wwids记录这设备的id号和对应的磁盘名称，对应的设备名称类似为/dev/mapper/mpatha,/dev/mapper/mpathb,/dev/mapper/mpath1,/dev/mapper/mpath2等，这些只是软链接，真正的设备为/dev/dm-*

multipath -F    --用来删除不使用的多路经设备
multipath -v2  --用来检测路径、多路径和设备映射
multipath -k    --通过交互式的方式来查看
multipath -ll    --查看多路径的状态
[root@db1 ~]# multipath -ll

mpath2 (26365653334643133) dm-0 Bigtera,VirtualStor
size=2.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:0 sdd 8:48  active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 14:0:0:0 sdl 8:176 active ready running
mpath1 (26235633739333036) dm-2 Bigtera,VirtualStor
size=1.5T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 11:0:0:0 sdf 8:80  active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 12:0:0:0 sdb 8:16  active ready running
mpath9 (26663643561363266) dm-12 Bigtera,VirtualStor
size=1.5T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 15:0:0:7 sds 65:32 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 16:0:0:7 sdw 65:96 active ready running

可以通过下面的方式修改RAC所需磁盘的属性，以下示例为RHEL6.4
/etc/udev/rules.d/12-dm-permissions.rules

ENV{DM_NAME}=="mpath2p1",OWNER:="ororgrid",GROUP:="dborgrid",MODE:="660"
ENV{DM_NAME}=="mpath3p1",OWNER:="ororgrid",GROUP:="dborgrid",MODE:="660"
ENV{DM_NAME}=="mpath1p1",OWNER:="ororgrid",GROUP:="dborgrid",MODE:="660"
ENV{DM_NAME}=="mpath1p2",OWNER:="ororgrid",GROUP:="dborgrid",MODE:="660"
ENV{DM_NAME}=="mpath1p3",OWNER:="ororgrid",GROUP:="dborgrid",MODE:="660"

B.在Linux上，EMC的Powerpath多路径软件安装后，通过/etc/rc.local修改后，磁盘属性会变化，需要通过下面的方式来解决

下面的示例是在RHEL6.4/etc/udev/rules.d/50-udev.rules

SUBSYSTEM=="block", KERNEL=="emcpowera", GROUP="asmadmin", OWNER="orgrid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="emcpowerb", GROUP="asmadmin", OWNER="orgrid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="emcpowerc", GROUP="asmadmin", OWNER="orgrid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="emcpowerd", GROUP="asmadmin", OWNER="orgrid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="emcpowere", GROUP="asmadmin", OWNER="orgrid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="emcpowerf", GROUP="asmadmin", OWNER="orgrid", MODE="0660"
SUBSYSTEM=="block", KERNEL=="emcpowerg", GROUP="asmadmin", OWNER="orgrid", MODE="0660"

EMC常用管理命令

https://community.emc.com/docs/DOC-18266

C.还可以采用raw device绑定，配置文件为/etc/sysconfig/rawdevices，不过这种方式现在使用的不多

/dev/raw/raw1 /dev/sda1

/dev/raw/raw2 /dev/sda2
/dev/raw/raw3 /dev/sda3
/dev/raw/raw4 /dev/sda4
/dev/raw/raw5 /dev/sda5
/dev/raw/raw6 /dev/sda6
/dev/raw/raw7 /dev/sda7
/dev/raw/raw8 /dev/sda8
chown orgrid:oinstall /dev/raw/rawn
chmod 640 /dev/raw/rawn
/sbin/service rawdevices restart
raw -qa
/dev/raw/raw1:  bound to major 3, minor 7
/dev/raw/raw2:  bound to major 3, minor 8

D.其他常用的写法

/etc/udev/rules.d/99-raw.rules

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/%P", RESULT=="3600805f30016c0e0ad41ffa8e6d90001", NAME="raw1", ACTION=="add|change",OWNER="root", GROUP="oinstall", MODE="0640"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/%P", RESULT=="3600805f30016c0e0ae51efbfb2930002", NAME="raw2", ACTION=="add|change",OWNER="oracle", GROUP="oinstall", MODE="0660"

/etc/udev/rules.d/99-oracle-asmdevices.rules

RHEL5/OL5/OL5

# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/%P", RESULT=="350002ac000010b52", NAME="vote1", ACTION=="add|change", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/%P", RESULT=="350002ac000020b52", NAME="vote2", ACTION=="add|change", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/%P", RESULT=="350002ac000030b52", NAME="vote3", ACTION=="add|change", OWNER="grid",GROUP="asmadmin", MODE="0660"
# /sbin/udevcontrol reload_rules
# /sbin/start_udev

RHEL6/OL6

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id i --whitelisted --device=/dev/$name", RESULT=="350002ac000010b52", NAME="vote1", ACTION=="add|change", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id i --whitelisted --device=/dev/$name", RESULT=="350002ac000020b52", NAME="vote2", ACTION=="add|change", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id i --whitelisted --device=/dev/$name", RESULT=="350002ac000030b52", NAME="vote3", ACTION=="add|change", OWNER="grid",GROUP="asmadmin", MODE="0660"
#/sbin/udevadm control --reload-rules
#/sbin/start_udev

Reference
http://docs.oracle.com/cd/E11882_01/install.112/e41961/storage.htm#CWLIN306
https://docs.oracle.com/cd/E52668_01/E54669/html/ol7-s1-devices.html
https://www.freedesktop.org/software/systemd/man/udevadm.html

https://www.freedesktop.org/software/systemd/man/udev.html

https://community.emc.com/docs/DOC-18266