目的

AZを跨いだ冗長構成（Active-Standby）
浮動ipでの運用でなく、ELBのヘルスチェックに依存させる
- 最近Updateされた機能により、運用が可能
  - Register or De-Register EC2 Instances for Your Load Balancer
オートディスカバリは無効にし、Zabbix-agent（アクティブ）にInternal-ELBを記述し、ノードの自動登録を行う

イメージ

環境

t2-micro(運用していくにはC4以上が好ましい)
Active/Standby構成（Standbyはサーバは起動しているが、F/Oのトリガーとなるプロセスを停止させている）
F/Oのトリガーはzabbix-serverの停止
クラスタ制御にはCorosync、リソース制御はPacemakerを使用
- Rundeck -構築（H/A）-

構築

インストール

$ sudo rpm -ivh http://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-2.4-1.el6.noarch.rpm
$ sudo yum install zabbix-server-mysql zabbix-web-mysql zabbix-web-japanese zabbix-agent

設定

MySQL

文字化けしないように事前にパラメータグループを作り、character_set_clientとcharacter_set_serverの値をutf-8にしておく。

$ mysql -h <rds-endpoint> -u <user> -p
mysql> grant all privileges on <dbname>.* to <user>@localhost identified by '<password>';

# 初期スキーマとデータをインポート
$ cd /usr/share/doc/zabbix-server-mysql-2.4.7/create
$ mysql -h <rds-endpoint> <db> -u <user> -p < schema.sql
$ mysql -h <rds-endpoint> <db> -u <user> -p < images.sql
$ mysql -h <rds-endpoint> <db> -u <user> -p < data.sql

zabbix-server

zabbix_server.conf

### Option: ListenPort
#       Listen port for trapper.
#
# Mandatory: no
# Range: 1024-32767
# Default
# ListenPort=10051

### Option: DBHost
#       Database host name.
#       If set to localhost, socket is used for MySQL.
#       If set to empty string, socket is used for PostgreSQL.
#
# Mandatory: no
# Default:
# DBHost=localhost
DBHost=<rds-endpoint>
### Option: DBName
#       Database name.
#       For SQLite3 path to database file must be provided. DBUser and DBPassword are ignored.
#
# Mandatory: yes
# Default:
# DBName=

DBName=<DBName>

### Option: DBSchema
#       Schema name. Used for IBM DB2 and PostgreSQL.
#
# Mandatory: no
# Default:
# DBSchema=

### Option: DBUser
#       Database user. Ignored for SQLite.
#
# Mandatory: no
# Default:
# DBUser=

DBUser=<user>

### Option: DBPassword
#       Database password. Ignored for SQLite.
#       Comment this line if no password is used.
#
# Mandatory: no
# Default:
# DBPassword=
DBPassword=********
### Option: DBSocket
#       Path to MySQL socket.
#
# Mandatory: no
# Default:
# DBSocket=/tmp/mysql.sock

DBSocket=/var/lib/mysql/mysql.sock

zabbix_agent.conf

### Option: Server
#       List of comma delimited IP addresses (or hostnames) of Zabbix servers.
#       Incoming connections will be accepted only from the hosts listed here.
#       If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
#
# Mandatory: no
# Default:
# Server=

Server=10.0.1.23,10.0.3.60

### Option: ListenPort
#       Agent will listen on this port for connections from the server.
#
# Mandatory: no
# Range: 1024-32767
# Default:
ListenPort=10050

### Option: ServerActive
#       List of comma delimited IP:port (or hostname:port) pairs of Zabbix servers for active checks.
#       If port is not specified, default port is used.
#       IPv6 addresses must be enclosed in square brackets if port for that host is specified.
#       If port is not specified, square brackets for IPv6 addresses are optional.
#       If this parameter is not specified, active checks are disabled.
#       Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=

ServerActive=<internal ELB Aレコード>or<internal ELB CNAME>

### Option: Hostname
#       Unique, case sensitive hostname.
#       Required for active checks and must match hostname as configured on the server.
#       Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

# Hostname=Zabbix server

### Option: HostnameItem
#       Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.
#       Does not support UserParameters or aliases.
#
# Mandatory: no
# Default:
HostnameItem=system.hostname

Apache/PHP

httpd.conf

$ sudo vim /etc/httpd/conf/httpd.conf
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/zabbix"

$ cd /var/www
$ sudo ln -s /usr/share/zabbix zabbix

$ sudo vim /etc/php.ini
post_max_size = 16M
max_execution_time = 300
max_input_time = 300
date.timezone = Asia/Tokyo
always_populate_raw_post_data = -1

フロントエンド

$ sudo service httpd start&&sudo chkconfig httpd on
$ sudo service zabbix-server start&&sudo chkconfig zabbix-server on
$ sudo service zabbix-agent start&&sudo chkconfig zabbix-agent on

自動登録アクション

Actionにのみ表示させるホスト情報を記述
- Conditions記述不要。Operationsは「ホスト追加」「テンプレートとリンク」

zabbix-agent

監視対象ノードに設定

インストール

$ sudo rpm -ivh http://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-2.4-1.el6.noarch.rpm
$ sudo yum -y install zabbix-agent

zabbix_agent.conf

### Option: SourceIP
#	Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=
SourceIP=10.0.3.6

### Option: Server
#       List of comma delimited IP addresses (or hostnames) of Zabbix servers.
#       Incoming connections will be accepted only from the hosts listed here.
#       If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
#
# Mandatory: no
# Default:
# Server=

Server=10.0.1.23,10.0.3.60

### Option: ListenPort
#       Agent will listen on this port for connections from the server.
#
# Mandatory: no
# Range: 1024-32767
# Default:
ListenPort=10050

### Option: ServerActive
#       List of comma delimited IP:port (or hostname:port) pairs of Zabbix servers for active checks.
#       If port is not specified, default port is used.
#       IPv6 addresses must be enclosed in square brackets if port for that host is specified.
#       If port is not specified, square brackets for IPv6 addresses are optional.
#       If this parameter is not specified, active checks are disabled.
#       Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=

ServerActive=<internal ELB Aレコード>or<internal ELB CNAME>

### Option: Hostname
#       Unique, case sensitive hostname.
#       Required for active checks and must match hostname as configured on the server.
#       Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

# Hostname=Zabbix server

### Option: HostnameItem
#       Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.
#       Does not support UserParameters or aliases.
#
# Mandatory: no
# Default:
HostnameItem=system.hostname

エラー

$ sudo service zabbix-agent start
Starting Zabbix agent:                                     [  OK  ]
$ sudo tail -f /var/log/zabbix/zabbix_agentd.log
  2709:20151228:150932.183 Starting Zabbix Agent [ip-10-0-3-6]. Zabbix 2.4.7 (revision 56694).
  2709:20151228:150932.183 using configuration file: /etc/zabbix/zabbix_agentd.conf
  2709:20151228:150932.183 agent #0 started [main process]
  2714:20151228:150932.183 agent #5 started [active checks #1]
  2713:20151228:150932.183 agent #4 started [listener #3]
  2712:20151228:150932.184 agent #3 started [listener #2]
  2711:20151228:150932.184 agent #2 started [listener #1]
  2710:20151228:150932.184 agent #1 started [collector]
  2714:20151228:150932.217 no active checks on server [******.elb.amazonaws.com:10051]: host [ip-10-0-3-6] not found

フロントエンドではノードの自動登録が成功しているように見えるが、Zabbix-serverとZabbix-agentとの通信が失敗している。
Agent interfacesで取得しているIPが不明
Internal-ELBのIPだった

$ dig internal-******.elb.amazonaws.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.42.amzn1 <<>> internal-******.elb.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38512
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;internal-******.elb.amazonaws.com. IN A

;; ANSWER SECTION:
internal-******.elb.amazonaws.com. 60 IN A 10.0.2.62
internal-******.elb.amazonaws.com. 60 IN A 10.0.2.58

;; Query time: 5 msec
;; SERVER: 10.0.0.2#53(10.0.0.2)
;; WHEN: Mon Dec 28 15:13:25 2015
;; MSG SIZE  rcvd: 116

対策としては、zabbix-agent.confのListenIPに自身のIPを記述することで、Agent interfacesで取得できるようになる。

### Option: ListenIP
#       List of comma delimited IP addresses that the agent should listen on.
#       First IP address is sent to Zabbix server if connecting to it to retrieve list of active checks.
#
# Mandatory: no
# Default:
ListenIP=10.0.3.6