LoginSignup
5
7

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@akito1986

apache + fluentd + TreasureData

Posted at

apacheのログをfluentdを用いてTreasureDataに入れてみたのでメモしておく。

背景

nginx + fluentd + TreasureDataでのapache版。

ほぼ上記のコピペだけど悪しからず。

以下環境情報。

  • ubuntu: 12.04
  • ruby: 2.1.1p76
  • gem: 2.2.2
  • apache2: Apache/2.2.22

TreasureDataのAPIキーは取得済みであることを前提とする。
TreasureDataを使ってみたを参照。

fluentd(td-agent)のインストール

以下のコマンドでfluentdの安定版であるtd-agentをインストールします。

install_td-agent
$ curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh
$ sudo td-agent --version
td-agent 0.10.45

DB作成

データの保存先のDBを作成しておく。

TD Tool BeltでTreasureDataにログインする。

login_td
$ td account -f
Enter your Treasure Data credentials.
Email: ${my_addr}
Password (typing will be hidden): ${my_passwd} 
Authenticated successfully.
Use 'td db:create <db_name>' to create a database.
$

DBを作成する。

create_db
$ td db:create apache

td-agentの設定

td-agentではデフォルトのapacheのフォーマットをサポートしてくれてるっぽいので設定は簡単。

以下のように/etc/td-agent/td-agent.confを編集。

td-agent.conf
# Tailing the Apache Log
<source>
  type tail
  path /var/log/apache2/access.log
  pos_file /var/log/td-agent/httpd-access.pos
  tag td.apache.access
  format apache2
</source>

# Treasure Data Input and Output
<match td.*.*>
  type tdlog
  apikey ${my_td_api_key}
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
  endpoint ${td_endpoint}
  flush_interval 10s
  use_ssl true
</match>

上記の意味ですが、source typeでtailを指定しており新規で追加されたものを対象としている。
source pathは監視対象のパス、source tagでタグ付けをしています。td.apache.accessはTreasureData上のapacheというDBのaccessというテーブルにデータを登録することを意味する。

matchの方ではendpointでTreasureDataのAPIのエンドポイントを指定する。AWSの場合だとhttps://api.treasuredata.comとなる。
flush_intervalでデータ送出間隔を指定します。

データインポート

まずはapache2を起動する。

start_apache2
$ sudo rm /var/log/apache2/access.log # 削除しておく。
$ sudo service apache2 restart

次にtd-agentの起動。

start_td-agent
$ sudo /etc/init.d/td-agent start

td-agentのログを見ると何やらエラーログをはいてる。

/var/log/td-agent/td-agent.log
2014-08-20 16:14:22 +0900 [error]: Permission denied - /var/log/apache2/access.log
  2014-08-20 16:14:22 +0900 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin/in_tail.rb:529:in `initialize'
  2014-08-20 16:14:22 +0900 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin/in_tail.rb:529:in `open'

apacheログの読み取り権限がないようなので付与してやる。

chmod
$ sudo ls -ld /var/log/apache2/
drwxr-x--- 2 root adm 4096 Aug 20 16:11 /var/log/apache2/
$ sudo chmod 755 /var/log/apache2/

念のためtd-agentの再起動。

td-agent_restart
$ /etc/init.d/td-agent restart
$ tail -f /var/log/td-agent/td-agent.log
</ROOT>
2014-08-20 16:19:43 +0900 [info]: adding source type="tail"
2014-08-20 16:19:43 +0900 [info]: adding source type="forward"
2014-08-20 16:19:43 +0900 [info]: adding source type="http"
2014-08-20 16:19:43 +0900 [info]: adding source type="debug_agent"
2014-08-20 16:19:43 +0900 [info]: adding match pattern="td.*.*" type="tdlog"
2014-08-20 16:19:43 +0900 [info]: adding match pattern="debug.**" type="stdout"
2014-08-20 16:19:43 +0900 [info]: following tail of /var/log/apache2/access.log
2014-08-20 16:19:43 +0900 [info]: listening fluent socket on 0.0.0.0:24224
2014-08-20 16:19:43 +0900 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
^C
$

さてこれで大丈夫なはず。apacheにアクセスしてみる。
念のため5回。

curl
$ curl http://localhost
$ curl http://localhost
$ curl http://localhost
$ curl http://localhost
$ curl http://localhost
$ sleep 10
check_log
$ sudo cat /var/log/apache2/access.log
127.0.0.1 - - [20/Aug/2014:16:22:26 +0900] "GET / HTTP/1.1" 200 432 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
127.0.0.1 - - [20/Aug/2014:16:22:27 +0900] "GET / HTTP/1.1" 200 432 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
127.0.0.1 - - [20/Aug/2014:16:22:28 +0900] "GET / HTTP/1.1" 200 432 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
127.0.0.1 - - [20/Aug/2014:16:22:28 +0900] "GET / HTTP/1.1" 200 432 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
127.0.0.1 - - [20/Aug/2014:16:22:29 +0900] "GET / HTTP/1.1" 200 432 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"

インポート確認

インポート確認。
5件のデータがインポートされていることがわかる。

check_import_data
$ td table:list apache
+----------+--------+------+-------+--------+---------------------------+---------------------------+----------------------------------------------------------------------------------------------------------+
| Database | Table  | Type | Count | Size   | Last import               | Last log timestamp        | Schema                                                                                                   |
+----------+--------+------+-------+--------+---------------------------+---------------------------+----------------------------------------------------------------------------------------------------------+
| apache   | access | log  | 5     | 0.0 GB | 2014-08-20 16:22:30 +0900 | 2014-08-20 16:22:29 +0900 | host:string, path:string, method:string, referer:string, code:long, agent:string, user:string, size:long |
+----------+--------+------+-------+--------+---------------------------+---------------------------+----------------------------------------------------------------------------------------------------------+
1 row in set

内容も確認してみる。

td_query
$ td query -w -t hive -d apache "SELECT * FROM access"
+-----------+------+--------+---------+------+-----------------------------------------------------------------------------------------------------+------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
| host      | path | method | referer | code | agent                                                                                               | user | size | v                                                                                                                                                                                                          | time       |
+-----------+------+--------+---------+------+-----------------------------------------------------------------------------------------------------+------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
| 127.0.0.1 | /    | GET    | null    | 200  | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | null | 432  | {"path":"/","code":"200","size":"432","method":"GET","host":"127.0.0.1","time":"1408519346","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1408519346 |
| 127.0.0.1 | /    | GET    | null    | 200  | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | null | 432  | {"path":"/","code":"200","size":"432","method":"GET","host":"127.0.0.1","time":"1408519347","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1408519347 |
| 127.0.0.1 | /    | GET    | null    | 200  | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | null | 432  | {"path":"/","code":"200","size":"432","method":"GET","host":"127.0.0.1","time":"1408519348","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1408519348 |
| 127.0.0.1 | /    | GET    | null    | 200  | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | null | 432  | {"path":"/","code":"200","size":"432","method":"GET","host":"127.0.0.1","time":"1408519348","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1408519348 |
| 127.0.0.1 | /    | GET    | null    | 200  | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | null | 432  | {"path":"/","code":"200","size":"432","method":"GET","time":"1408519349","host":"127.0.0.1","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1408519349 |
+-----------+------+--------+---------+------+-----------------------------------------------------------------------------------------------------+------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
5 rows in set

ふむふむ、ちゃんと各カラムにデータが入ってるっぽい。

おしまい。

5
7
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
5
7

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?