はじめに

EC2 インスタンスは、デフォルトでいくつかの Metrics をモニタリングできます。デフォルトのものでは、メモリや Disk の使用量などの Metrics は取得できません。CloudWatch Agent を、新たにインストールして設定することで、メモリや Disk を含む詳細な Metrics を取得出来ます。取得できるものは次の Document に一覧されています。

この記事では、Amazon Linux ２に、CloudWatch Agent の導入方法を備忘録的にメモします。

通常の Monitoring

まずはじめに、何も気にせずに EC2 インスタンスを作ると、EC2 インスタンスの Monitoring の Tab に、該当インスタンスのいくつかの Metrics が見えます。ただ、メモリや Disk の使用量などのメトリクスは取得できていません。

CPU utilization
Status check failed
Network in
Network out
Disk reads
Disk write
CPU credit usage
CPU credit balance

CloudWatch Agent Install

CloudWatch Agent を Install します。

sudo yum install amazon-cloudwatch-agent

依存関係

============================================================================================================================
 Package                              Arch                Version                             Repository               Size
============================================================================================================================
Installing:
 amazon-cloudwatch-agent              x86_64              1.247345.35-1.amzn2                 amzn2-core               27 M

Transaction Summary
============================================================================================================================

設定ファイルが自動生成されています

[ec2-user@ip-10-1-1-91 etc]$ ls -la /opt/aws/amazon-cloudwatch-agent/etc
total 4
drwxr-xr-x 3 root root  65 Feb 12 14:07 .
drwxr-xr-x 7 root root 140 Feb 12 14:07 ..
drwxr-xr-x 2 root root   6 Sep 24 21:43 amazon-cloudwatch-agent.d
-rw-r--r-- 1 root root 925 Sep 24 21:43 common-config.toml
[ec2-user@ip-10-1-1-91 etc]$

中身はこんな感じです

[ec2-user@ip-10-1-1-91 etc]$ cat /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
# This common-config is used to configure items used for both ssm and cloudwatch access


## Configuration for shared credential.
## Default credential strategy will be used if it is absent here:
##      Instance role is used for EC2 case by default.
##      AmazonCloudWatchAgent profile is used for onPremise case by default.
# [credentials]
#    shared_credential_profile = "{profile_name}"
#    shared_credential_file = "{file_name}"


## Configuration for proxy.
## System-wide environment-variable will be read if it is absent here.
## i.e. HTTP_PROXY/http_proxy; HTTPS_PROXY/https_proxy; NO_PROXY/no_proxy
## Note: system-wide environment-variable is not accessible when using ssm run-command.
## Absent in both here and environment-variable means no proxy will be used.
# [proxy]
#    http_proxy = "{http_url}"
#    https_proxy = "{https_url}"
#    no_proxy = "{domain}"

# [ssl]
#    ca_bundle_path = "{ca_bundle_file_path}"
[ec2-user@ip-10-1-1-91 etc]$

IAM Role 作成

CloudWatch Agent が EC2 インスタンス内の Metrics 情報を、CloudWatch に書き込むための IAM Role を作成します。

EC2 を選びます

次の名前で検索して選択します。

CloudWatchAgentServerPolicy

適当な名前を入れます

適当な名前をいれて、Create role をします

作成完了

IAM Role を EC2 インスタンスにアタッチ

EC２インスタンスに、作成した IAM Role をアタッチします。

Save

CloudWatch Agent 設定

CloudWatch でどのように Metrics を取得するか設定をしていきます。wizard が用意されているので、実行します。

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Linux なので 1

[ec2-user@ip-10-1-1-91 bin]$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
=============================================================
= Welcome to the AWS CloudWatch Agent Configuration Manager =
=============================================================
On which OS are you planning to use the agent?
1. linux
2. 2. windows
3. default choice: [1]:
4. 1

EC2 なので 1

Trying to fetch the default region based on ec2 metadata...
Are you using EC2 or On-Premises hosts?
1. EC2
2. 2. On-Premises
3. default choice: [1]:
4. 1

Default で 1

Which user are you planning to run the agent?
1. root
2. 2. cwagent
3. 3. others
4. default choice: [1]:
5. 1

StatsD daemon を起動しておく

Do you want to turn on StatsD daemon?
1. yes
2. 2. no
3. default choice: [1]:
4. 1

Default のまま

Which port do you want StatsD daemon to listen to?
default choice: [8125]

1 で Default のまま

What is the collect interval for StatsD daemon?
1. 10s
2. 2. 30s
3. 3. 60s
4. default choice: [1]:
5. 1

60 秒おきに集約

What is the aggregation interval for metrics collected by StatsD daemon?
1. Do not aggregate
2. 2. 10s
3. 3. 30s
4. 4. 60s
5. default choice: [4]:
6. 4

CollectD を有効

Do you want to monitor metrics from CollectD?
1. yes
2. 2. no
3. default choice: [1]:
4. 1

Yes

Do you want to monitor any host metrics? e.g. CPU, memory, etc.
1. yes
2. 2. no
3. default choice: [1]:
4. 1

Yes

Do you want to monitor cpu metrics per core? Additional CloudWatch charges may apply.
1. yes
2. 2. no
3. default choice: [1]:
4. 1

Yes

Do you want to add ec2 dimensions (ImageId, InstanceId, InstanceType, AutoScalingGroupName) into all of your metrics if the info is available?
1. yes
2. 2. no
3. default choice: [1]:
4. 1

Would you like to collect your metrics at high resolution (sub-minute resolution)? This enables sub-minute resolution for all metrics, but you can customize for specific metrics in the output json file.
1. 1s
2. 2. 10s
3. 3. 30s
4. 4. 60s
5. default choice: [4]:
6. 4

取得する Metrics の多さ

Advanced が最も多い

Which default metrics config do you want?
1. Basic
2. 2. Standard
3. 3. Advanced
4. 4. None
5. default choice: [1]:
6. 3

保存

Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.
1. yes
2. 2. no
3. default choice: [1]:
4. 1

ほかに Agent はないので 2

Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration?
1. yes
2. 2. no
3. default choice: [2]:
4. 2

log も取得する

Do you want to monitor any log files?
1. yes
2. 2. no
3. default choice: [1]:
4. 1

/var/log/messages を取得しておく

Log file path:
/var/log/messages

log group name

Log group name:
default choice: [messages]
messages

Log Stream Name

default choice: [{instance_id}]
testagent-messages-stream

他に監視したいファイルはないので 2

Do you want to specify any additional log files to monitor?
1. yes
2. 2. no
3. default choice: [1]:
4. 2

保存しないので2

Do you want to store the config in the SSM parameter store?
1. yes
2. 2. no
3. default choice: [1]:
4. 2

設定ファイルが生成されている

[ec2-user@ip-10-1-1-91 bin]$ cat /opt/aws/amazon-cloudwatch-agent/bin/config.json
{
        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "root"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/messages",
                                                "log_group_name": "messages",
                                                "log_stream_name": "testagent-messages-stream"
                                        }
                                ]
                        }
                }
        },
        "metrics": {
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "collectd": {
                                "metrics_aggregation_interval": 60
                        },
                        "cpu": {
                                "measurement": [
                                        "cpu_usage_idle",
                                        "cpu_usage_iowait",
                                        "cpu_usage_user",
                                        "cpu_usage_system"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ],
                                "totalcpu": false
                        },
                        "disk": {
                                "measurement": [
                                        "used_percent",
                                        "inodes_free"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "diskio": {
                                "measurement": [
                                        "io_time",
                                        "write_bytes",
                                        "read_bytes",
                                        "writes",
                                        "reads"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        },
                        "netstat": {
                                "measurement": [
                                        "tcp_established",
                                        "tcp_time_wait"
                                ],
                                "metrics_collection_interval": 60
                        },
                        "statsd": {
                                "metrics_aggregation_interval": 60,
                                "metrics_collection_interval": 10,
                                "service_address": ":8125"
                        },
                        "swap": {
                                "measurement": [
                                        "swap_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        }
                }
        }
}

Collectd のインストール

wizard で collectd を使う設定をしたので、インストールします

sudo amazon-linux-extras install collectd

CloudWatch Agent の実行

次のコマンドで、CloudWatch Agent を実行します。

cd /opt/aws/amazon-cloudwatch-agent/bin/
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:config.json

実行例

[ec2-user@ip-10-1-1-91 bin]$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:config.json
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
2021/02/12 15:22:58 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...
Valid Json input schema.
I! Detecting runasuser...
No csm configuration found.
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
Created symlink from /etc/systemd/system/multi-user.target.wants/amazon-cloudwatch-agent.service to /etc/systemd/system/amazon-cloudwatch-agent.service.
Redirecting to /bin/systemctl restart amazon-cloudwatch-agent.service
[ec2-user@ip-10-1-1-91 bin]$

これにより、Systemd 上でも Service 起動されていて、自動起動設定が有効になっています

[ec2-user@ip-10-1-1-91 bin]$ systemctl status amazon-cloudwatch-agent
● amazon-cloudwatch-agent.service - Amazon CloudWatch Agent
   Loaded: loaded (/etc/systemd/system/amazon-cloudwatch-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-02-12 15:22:58 UTC; 47s ago
 Main PID: 1266 (amazon-cloudwat)
   CGroup: /system.slice/amazon-cloudwatch-agent.service
           └─1266 /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml -envconfig /opt/aws/amazon-cloudwatch-agent/e...

Feb 12 15:22:58 ip-10-1-1-91.ap-northeast-1.compute.internal systemd[1]: Started Amazon CloudWatch Agent.
Feb 12 15:22:58 ip-10-1-1-91.ap-northeast-1.compute.internal systemd[1]: Starting Amazon CloudWatch Agent...
Feb 12 15:22:58 ip-10-1-1-91.ap-northeast-1.compute.internal start-amazon-cloudwatch-agent[1266]: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot r...ping it.
Feb 12 15:22:58 ip-10-1-1-91.ap-northeast-1.compute.internal start-amazon-cloudwatch-agent[1266]: Valid Json input schema.
Feb 12 15:22:58 ip-10-1-1-91.ap-northeast-1.compute.internal start-amazon-cloudwatch-agent[1266]: I! Detecting runasuser...
Hint: Some lines were ellipsized, use -l to show in full.
[ec2-user@ip-10-1-1-91 bin]$
[ec2-user@ip-10-1-1-91 bin]$