今日は Oracle Real Application Clusters (RAC) の VIP、Scan Listnerが起動できず、クライアント接続ができない事象に遭遇したので対処してみてみます。
■ Clusterware ステータス 確認
Oracle Grid Infrastructure(GI) の Clusterwareリソースのステータスおよび構成情報を取得し確認すると VIP が起動せず、それに依存する SCAN Listner なども OFFLINE の状態です。
[grid@db-tokyo1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.COMMONSTORE.advm
ONLINE ONLINE db-tokyo1 STABLE
ONLINE ONLINE db-tokyo2 STABLE
ora.LISTENER.lsnr
ONLINE OFFLINE db-tokyo1 STABLE
ONLINE OFFLINE db-tokyo2 STABLE
ora.chad
ONLINE ONLINE db-tokyo1 STABLE
OFFLINE OFFLINE db-tokyo2 STABLE
ora.data.commonstore.acfs
ONLINE ONLINE db-tokyo1 mounted on /opt/orac
le/dcs/commonstore,S
TABLE
ONLINE ONLINE db-tokyo2 mounted on /opt/orac
le/dcs/commonstore,S
TABLE
ora.net1.network
ONLINE OFFLINE db-tokyo1 STABLE
ONLINE OFFLINE db-tokyo2 STABLE
ora.ons
ONLINE OFFLINE db-tokyo1 STABLE
ONLINE OFFLINE db-tokyo2 STABLE
ora.proxy_advm
ONLINE ONLINE db-tokyo1 STABLE
ONLINE ONLINE db-tokyo2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 STABLE
2 ONLINE ONLINE db-tokyo2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 STABLE
2 ONLINE ONLINE db-tokyo2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE OFFLINE STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE OFFLINE STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE OFFLINE STABLE
ora.RECO.dg(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 STABLE
2 ONLINE ONLINE db-tokyo2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 Started,STABLE
2 ONLINE ONLINE db-tokyo2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 STABLE
2 ONLINE ONLINE db-tokyo2 STABLE
ora.cdb_tokyo.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cvu
1 ONLINE OFFLINE STABLE
ora.db-tokyo1.vip
1 ONLINE OFFLINE STABLE
ora.db-tokyo2.vip
1 ONLINE OFFLINE STABLE
ora.qosmserver
1 ONLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE OFFLINE STABLE
ora.scan2.vip
1 ONLINE OFFLINE STABLE
ora.scan3.vip
1 ONLINE OFFLINE STABLE
--------------------------------------------------------------------------------
■ VIPリソース起動
srvctl start vip で起動してみてみます。
CRS-5008: Invalid attribute value: ens3 for the network interface というエラーが出ています。
● Node1
[grid@db-tokyo1 ~]$ srvctl start vip -vip db-tokyo1
PRCR-1079 : Failed to start resource ora.db-tokyo1.vip
CRS-5017: The resource action "ora.net1.network start" encountered the following error:
CRS-5008: Invalid attribute value: ens3 for the network interface
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/db-tokyo1/crs/trace/crsd_orarootagent_root.trc".
CRS-2674: Start of 'ora.net1.network' on 'db-tokyo1' failed
CRS-5017: The resource action "ora.net1.network start" encountered the following error:
CRS-5008: Invalid attribute value: ens3 for the network interface
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/db-tokyo2/crs/trace/crsd_orarootagent_root.trc".
CRS-2674: Start of 'ora.net1.network' on 'db-tokyo2' failed
CRS-2632: There are no more servers to try to place resource 'ora.db-tokyo1.vip' on that would satisfy its placement policy
● Node2
[grid@db-tokyo2 ~]$ srvctl start vip -vip db-tokyo2
PRCR-1079 : Failed to start resource ora.db-tokyo2.vip
CRS-5017: The resource action "ora.net1.network start" encountered the following error:
CRS-5008: Invalid attribute value: ens3 for the network interface
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/db-tokyo2/crs/trace/crsd_orarootagent_root.trc".
CRS-2674: Start of 'ora.net1.network' on 'db-tokyo2' failed
CRS-2632: There are no more servers to try to place resource 'ora.db-tokyo2.vip' on that would satisfy its placement policy
■ OS Network Interface確認
ifconfigで確認してみてみます。
VIP起動するサービス用IFの ens3は起動していて ssh接続もできるので、特に問題なさそうです。
● Node1
[grid@db-tokyo1 ~]$ ifconfig -a
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 10.10.1.10 netmask 255.255.255.224 broadcast 10.10.1.31
ether 02:00:17:03:12 txqueuelen 1000 (Ethernet)
RX packets 170230 bytes 1116295467 (1.0 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 100517 bytes 140217187 (133.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 192.168.16.18 netmask 255.255.255.0 broadcast 192.168.16.255
ether 02:00:17:00:55 txqueuelen 1000 (Ethernet)
RX packets 52669 bytes 74312772 (70.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 58455 bytes 145003999 (138.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 11955 bytes 5919795 (5.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11955 bytes 5919795 (5.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
● Node2
[grid@db-tokyo2 ~]$ ifconfig -a
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 10.10.1.4 netmask 255.255.255.224 broadcast 10.10.1.31
ether 02:00:17:02:48 txqueuelen 1000 (Ethernet)
RX packets 396946 bytes 1472978705 (1.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 249472 bytes 741036999 (706.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 192.168.16.19 netmask 255.255.255.0 broadcast 192.168.16.255
ether 02:00:17:02:a4 txqueuelen 1000 (Ethernet)
RX packets 60078 bytes 149204245 (142.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 54843 bytes 76570483 (73.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 55376 bytes 225465544 (215.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 55376 bytes 225465544 (215.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
■ ora.net1.networkリソース設定確認
CRS-5008: Invalid attribute value: ens3 for the network interface というエラーをたよりに、GIのネットワーク関連が設定されてる ora.net1.networkリソースのパラメータ内容を確認してみようと思います。
crsctl status resourceで確認してみると ens3インターフェースのNETMASKが ifconfig と異なる値設定されています。
・ifconfig: netmask 255.255.255.224
・ora.net1.network: USR_ORA_NETMASK=255.255.255.240
[root@db-tokyo1 ~]# su - grid
[grid@db-tokyo1 ~]$ crsctl status resource ora.net1.network -p | grep USR_ORA
USR_ORA_AUTO=static
USR_ORA_ENV=
USR_ORA_IF=ens3
USR_ORA_NETMASK=255.255.255.240
USR_ORA_SUBNET=10.10.1.0
[grid@db-tokyo1 ~]$ crsctl status resource ora.net1.network -p
NAME=ora.net1.network
TYPE=ora.network.type
ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:grid:r-x
ACTIONS=ipsec_setup,user:"grid"
ACTION_SCRIPT=
ACTION_TIMEOUT=60
ADDRESS_TYPE=IPV4
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CHECK_INTERVAL=1
CHECK_TIMEOUT=0
CLEAN_TIMEOUT=60
CSS_CRITICAL=no
DELETE_TIMEOUT=60
DESCRIPTION=Oracle Network resource
ENABLED=1
IGNORE_TARGET_ON_FAILURE=no
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
NLS_LANG=
NODE_VIPLESS=false
OFFLINE_CHECK_INTERVAL=60
PING_TARGET=
RESOURCE_GROUP=
RESTART_ATTEMPTS=5
RESTART_DELAY=0
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=ora.hub.category
START_CONCURRENCY=0
START_DEPENDENCIES=
START_TIMEOUT=0
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=
STOP_TIMEOUT=0
TARGET_DEFAULT=atleastone
TYPE_VERSION=3.3
UPTIME_THRESHOLD=1d
USER_WORKLOAD=no
USR_ORA_AUTO=static
USR_ORA_ENV=
USR_ORA_IF=ens3
USR_ORA_NETMASK=255.255.255.240
USR_ORA_SUBNET=10.10.1.0
WORKLOAD_CPU=0
WORKLOAD_CPU_CAP=0
WORKLOAD_MEMORY_MAX=0
WORKLOAD_MEMORY_TARGET=0
■ ora.net1.network修正
GIのNetowrkリソース ora.net1.network を ens3インターフェース の NETMASK 255.255.255.224 と同じ設定に修正してみてみます。
srvctl modify network は rootユーザーで実行します。
構文
# srvctl modify network -k <network_number> -S <subnet>/<netmask>/<interface1>
[root@db-tokyo1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl modify network -k 1 -S 10.10.1.0/255.255.255.224/ens3
[root@db-tokyo1 ~]# su - grid
[grid@db-tokyo1 ~]$ crsctl status resource ora.net1.network -p | grep USR_ORA
USR_ORA_AUTO=static
USR_ORA_ENV=
USR_ORA_IF=ens3
USR_ORA_NETMASK=255.255.255.224
USR_ORA_SUBNET=10.10.1.0
■ OS再起動
全Node再起動
Node1
[root@db-tokyo1 ~]# shutdown -r now
Node2
[root@db-tokyo2 ~]# shutdown -r now
■ 正常稼働確認
正常に起動しました ＼(^o^)／
[grid@db-tokyo1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.COMMONSTORE.advm
ONLINE ONLINE db-tokyo1 STABLE
ONLINE ONLINE db-tokyo2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE db-tokyo1 STABLE
ONLINE ONLINE db-tokyo2 STABLE
ora.chad
ONLINE ONLINE db-tokyo1 STABLE
ONLINE ONLINE db-tokyo2 STABLE
ora.data.commonstore.acfs
ONLINE ONLINE db-tokyo1 mounted on /opt/orac
le/dcs/commonstore,S
TABLE
ONLINE ONLINE db-tokyo2 mounted on /opt/orac
le/dcs/commonstore,S
TABLE
ora.net1.network
ONLINE ONLINE db-tokyo1 STABLE
ONLINE ONLINE db-tokyo2 STABLE
ora.ons
ONLINE ONLINE db-tokyo1 STABLE
ONLINE ONLINE db-tokyo2 STABLE
ora.proxy_advm
ONLINE ONLINE db-tokyo1 STABLE
ONLINE ONLINE db-tokyo2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 STABLE
2 ONLINE ONLINE db-tokyo2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 STABLE
2 ONLINE ONLINE db-tokyo2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE db-tokyo1 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE db-tokyo2 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE db-tokyo2 STABLE
ora.RECO.dg(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 STABLE
2 ONLINE ONLINE db-tokyo2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 Started,STABLE
2 ONLINE ONLINE db-tokyo2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE db-tokyo1 STABLE
2 ONLINE ONLINE db-tokyo2 STABLE
ora.cdb_tokyo.db
1 ONLINE ONLINE db-tokyo1 Open,HOME=/u01/app/o
racle/product/19.0.0
.0/dbhome_1,STABLE
2 ONLINE ONLINE db-tokyo2 Open,HOME=/u01/app/o
racle/product/19.0.0
.0/dbhome_1,STABLE
ora.cvu
1 ONLINE ONLINE db-tokyo2 STABLE
ora.db-tokyo1.vip
1 ONLINE ONLINE db-tokyo1 STABLE
ora.db-tokyo2.vip
1 ONLINE ONLINE db-tokyo2 STABLE
ora.qosmserver
1 ONLINE ONLINE db-tokyo2 STABLE
ora.scan1.vip
1 ONLINE ONLINE db-tokyo1 STABLE
ora.scan2.vip
1 ONLINE ONLINE db-tokyo2 STABLE
ora.scan3.vip
1 ONLINE ONLINE db-tokyo2 STABLE
--------------------------------------------------------------------------------
■ 原因
VCN Subnet CIDRサイズを 255.255.255.240(/28)から 255.255.255.224(/27)へ拡張変更したのだそうな、
OSのens3インターフェースのNetowrk設定は更新されたが、GIの Networkリソースは更新しなかったので、GIのVIPが起動できなくなった、ということでした。
■ 参考
・キュメント: Oracle Clusterware制御(CRSCTL)ユーティリティ・リファレンス
・My Oracle Support: Unable To Start ora.net2.network with CRS-2672 CRS-2674 and CRS-5008: Invalid attribute value: en4 for the network interface (Doc ID 1548049.1)