KinesisAgentインストールと動作確認
KinesisFirehoseを使うのでProducerにKinesisエージェントを使ってみる。メモとして記録。検証時のOSはAmazon Linux 2。
AmazonLinux方式のインストール
$ sudo yum update
$ sudo yum install aws-kinesis-agent
Loaded plugins: langpacks, priorities, update-motd
104 packages excluded due to repository priority protections
No package aws-kinesis-agent available.
AmazonLinux方式うまくいかない。。
調査する余裕もないのでS3にあるほうをインストールする
$ sudo yum install –y https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn1.noarch.rpm
======================================================================================================================
Package Arch Version Repository Size
======================================================================================================================
Installing:
aws-kinesis-agent noarch 1.1.3-1.amzn1 /aws-kinesis-agent-latest.amzn1.noarch 18 M
Installing for dependencies:
alsa-lib x86_64 1.1.3-3.amzn2 amzn2-core 425 k
copy-jdk-configs noarch 2.2-3.amzn2 amzn2-core 19 k
geronimo-jms noarch 1.1.1-19.amzn2 amzn2-core 31 k
giflib x86_64 4.1.6-9.amzn2 amzn2-core 40 k
java-1.8.0-openjdk x86_64 1:1.8.0.161-0.b14.amzn2 amzn2-core 242 k
java-1.8.0-openjdk-headless x86_64 1:1.8.0.161-0.b14.amzn2 amzn2-core 32 M
javamail noarch 1.4.6-8.amzn2 amzn2-core 757 k
javapackages-tools noarch 3.4.1-11.amzn2 amzn2-core 73 k
libXcomposite x86_64 0.4.4-4.1.amzn2 amzn2-core 22 k
libXfont x86_64 1.5.2-1.amzn2 amzn2-core 152 k
libXi x86_64 1.7.9-1.amzn2 amzn2-core 41 k
libXrender x86_64 0.9.10-1.amzn2 amzn2-core 26 k
libXtst x86_64 1.2.3-1.amzn2 amzn2-core 20 k
libfontenc x86_64 1.1.3-3.amzn2 amzn2-core 31 k
lksctp-tools x86_64 1.0.17-2.amzn2 amzn2-core 88 k
log4j noarch 1.2.17-16.amzn2 amzn2-core 444 k
python-javapackages noarch 3.4.1-11.amzn2 amzn2-core 31 k
python-lxml x86_64 3.2.1-4.amzn2 amzn2-core 779 k
ttmkfdir x86_64 3.0.9-42.amzn2 amzn2-core 50 k
tzdata-java noarch 2018c-1.amzn2 amzn2-core 184 k
xorg-x11-font-utils x86_64 1:7.5-20.amzn2 amzn2-core 87 k
xorg-x11-fonts-Type1 noarch 7.5-9.amzn2 amzn2-core 521 k
Transaction Summary
======================================================================================================================
Install 1 Package (+22 Dependent packages)
設定ファイルを確認。デフォルトだとこんな状態。
$ ls -l /etc/aws-kinesis/agent.json
-rw-r--r-- 1 root root 338 Dec 22 09:17 /etc/aws-kinesis/agent.json
$ cat /etc/aws-kinesis/agent.json
{
"cloudwatch.emitMetrics": true,
"kinesis.endpoint": "",
"firehose.endpoint": "",
"flows": [
{
"filePattern": "/tmp/app.log*",
"kinesisStream": "yourkinesisstream",
"partitionKeyOption": "RANDOM"
},
{
"filePattern": "/tmp/app.log*",
"deliveryStream": "yourdeliverystream"
}
]
}
- エージェントがデータを集めるファイル ("filePattern")
エージェントがデータを送信するストリーム ("kinesisStream")
Kinesis側の設定が完了してから設定する自動起動を有効にする
$ sudo chkconfig aws-kinesis-agent on
- 手動で起動する
$ sudo service aws-kinesis-agent start
Starting aws-kinesis-agent (via systemctl): [ OK ]
- ログファイルが作られているので確認
$ cat /var/log/aws-kinesis-agent/aws-kinesis-agent.log