LoginSignup
9
9

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@akito1986

nginx + fluentd + TreasureData

Posted at

nginxのログをfluentdを用いてTreasureDataに入れてみたのでメモしておく。

背景

TreasureDataを使ってみたで書いたのだが、要はログの管理が面倒で、容量とかのログ管理をせずにログを貯めたい！ってのがやりたいこと。

それで縁あって上のやりたいことがTreasureDataを使ってできるかを確かめてみたって感じ。

以下環境情報。

  • ubuntu: 12.04
  • ruby: 2.1.2p95
  • gem: 2.2.2
  • nginx: nginx/1.1.19

TreasureDataのAPIキーは取得済みであることを前提とする。
TreasureDataを使ってみたを参照。

fluentd(td-agent)のインストール

以下のコマンドでfluentdの安定版であるtd-agentをインストールします。

install_td-agent
$ curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-precise.sh | sh
$ sudo td-agent --version
td-agent 0.10.50

DB作成

データの保存先のDBを作成しておく。

TD Tool BeltでTreasureDataにログインする。

login_td
$ td account -f
Enter your Treasure Data credentials.
Email: ${my_addr}
Password (typing will be hidden): ${my_passwd} 
Authenticated successfully.
Use 'td db:create <db_name>' to create a database.
$

DBを作成する。

create_db
$ td db:create nginx

td-agentの設定

td-agentではデフォルトのnginxのフォーマットをサポートしてくれてるっぽいので設定は簡単。

以下のように/etc/td-agent/td-agent.confを編集。

td-agent.conf
# Tailing the Nginx Log
<source>
  type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx-access.pos
  tag td.nginx.access
  format nginx
</source>

# Treasure Data Input and Output
<match td.*.*>
  type tdlog
  apikey ${my_td_api_key}
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
  endpoint ${td_endpoint}
  flush_interval 10s
  use_ssl true
</match>

上記の意味ですが、source typeでtailを指定しており新規で追加されたものを対象としている。
source pathは監視対象のパス、source tagでタグ付けをしています。td.nginx.accessはTreasureData上のnginxというDBのaccessというテーブルにデータを登録することを意味する。

matchの方ではendpointでTreasureDataのAPIのエンドポイントを指定する。AWSの場合だとhttps://api.treasuredata.comとなる。
flush_intervalでデータ送出間隔を指定します。

データインポート

まずはnginxを起動する。

start_nginx
$ sudo rm /var/log/nginx/access.log # 削除しておく。
$ sudo service nginx start

次にtd-agentの起動。

start_td-agent
$ sudo /etc/init.d/td-agent start

nginxにアクセスしてみる。
念のため5回。

curl
$ curl http://localhost
$ curl http://localhost
$ curl http://localhost
$ curl http://localhost
$ curl http://localhost
$ sleep 10
check_log
$ sudo cat /var/log/nginx/access.log
127.0.0.1 - - [14/Aug/2014:01:26:09 +0900] "GET / HTTP/1.1" 404 169 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
127.0.0.1 - - [14/Aug/2014:01:26:11 +0900] "GET / HTTP/1.1" 404 169 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
127.0.0.1 - - [14/Aug/2014:01:26:12 +0900] "GET / HTTP/1.1" 404 169 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
127.0.0.1 - - [14/Aug/2014:01:26:12 +0900] "GET / HTTP/1.1" 404 169 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
127.0.0.1 - - [14/Aug/2014:01:26:13 +0900] "GET / HTTP/1.1" 404 169 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"

インポート確認

インポート確認。
5件のデータがインポートされていることがわかる。

check_import_data
$ td table:list nginx
+----------+--------+------+-------+--------+---------------------------+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| Database | Table  | Type | Count | Size   | Last import               | Last log timestamp        | Schema                                                                                                                      |
+----------+--------+------+-------+--------+---------------------------+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| nginx    | access | log  | 5     | 0.0 GB | 2014-08-14 01:26:13 +0900 | 2014-08-14 01:26:13 +0900 | host:string, path:string, method:string, referer:string, code:string, remote:string, agent:string, user:string, size:string |
+----------+--------+------+-------+--------+---------------------------+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+

内容も確認してみる。

td_query
$ td query -w -t hive -d nginx "SELECT * FROM access"
+------+------+--------+---------+------+-----------+-----------------------------------------------------------------------------------------------------+------+------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
| host | path | method | referer | code | remote    | agent                                                                                               | user | size | v                                                                                                                                                                                                                                                | time       |
+------+------+--------+---------+------+-----------+-----------------------------------------------------------------------------------------------------+------+------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
| -    | /    | GET    | -       | 404  | 127.0.0.1 | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | -    | 169  | {"host":"-","user":"-","path":"/","referer":"-","method":"GET","code":"404","size":"169","time":"1407947169","remote":"127.0.0.1","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1407947169 |
| -    | /    | GET    | -       | 404  | 127.0.0.1 | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | -    | 169  | {"host":"-","user":"-","path":"/","referer":"-","method":"GET","code":"404","size":"169","remote":"127.0.0.1","time":"1407947171","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1407947171 |
| -    | /    | GET    | -       | 404  | 127.0.0.1 | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | -    | 169  | {"host":"-","user":"-","path":"/","referer":"-","method":"GET","code":"404","size":"169","remote":"127.0.0.1","time":"1407947172","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1407947172 |
| -    | /    | GET    | -       | 404  | 127.0.0.1 | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | -    | 169  | {"host":"-","user":"-","path":"/","referer":"-","method":"GET","code":"404","size":"169","remote":"127.0.0.1","time":"1407947172","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1407947172 |
| -    | /    | GET    | -       | 404  | 127.0.0.1 | curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 | -    | 169  | {"host":"-","user":"-","path":"/","referer":"-","method":"GET","code":"404","size":"169","remote":"127.0.0.1","time":"1407947173","agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"} | 1407947173 |
+------+------+--------+---------+------+-----------+-----------------------------------------------------------------------------------------------------+------+------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
5 rows in set

ふむふむ、ちゃんと各カラムにデータが入ってるっぽい。

次回は独自フォーマットのログのインポート設定について記載するつもり。

おしまい。

9
9
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
9
9

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?