golangで、時系列データベース「InfluxDB」を試してみる

今回、時系列データベース”InfluxDB”の基本動作を試してみました。

⬛︎ InfluxDB環境構築

実際の構築にあたって、こちらのInstallationページを参考にしました。

(1) Linux環境準備

• Ubunts環境を確認する

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"

• golangをインストールする

$ vi $HOME/.profile

...(snip)
export GOPATH=$HOME/golang
export PATH=$GOPATH/bin:/usr/local/go/bin:$PATH

$ wget --no-check-certificate https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.6.2.linux-amd64.tar.gz
$ mkdir $HOME/golang
$ source .profile

$ go version
go version go1.6.2 linux/amd64

(2) InfluxDB環境構築

• InfluxDBをインストールする

$ sudo apt-get update
$ curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
$ echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

$ sudo apt-get update && sudo apt-get install influxdb

• InfluxDBを起動する

$ sudo service influxdb start
$ sudo service influxdb status
● influxdb.service - InfluxDB is an open-source, distributed, time series database
   Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2016-07-17 17:29:40 JST; 17s ago
     Docs: https://docs.influxdata.com/influxdb/
 Main PID: 1932 (sh)
    Tasks: 8 (limit: 512)
   Memory: 7.8M
      CPU: 53ms
   CGroup: /system.slice/influxdb.service
           ├─1932 /bin/sh -c /usr/bin/influxd -config /etc/influxdb/influxdb.conf  >>/dev/null 2>>/var/log/influxdb/influxd.log
           └─1934 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

Jul 17 17:29:40 influxdb systemd[1]: Started InfluxDB is an open-source, distributed, time series database.

• InfluxDBの動作を確認する

$ influxd

 8888888           .d888 888                   8888888b.  888888b.
   888            d88P"  888                   888  "Y88b 888  "88b
   888            888    888                   888    888 888  .88P
   888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
   888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
   888   888  888 888    888 888  888   X88K   888    888 888    888
   888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
 8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"

[run] 2016/07/17 17:32:11 InfluxDB starting, version 0.13.0, branch 0.13, commit e57fb88a051ee40fd9277094345fbd47bb4783ce
[run] 2016/07/17 17:32:11 Go version go1.6.2, GOMAXPROCS set to 1
[run] 2016/07/17 17:32:11 Using configuration at: /etc/influxdb/influxdb.conf
run: open server: listen: listen tcp :8088: bind: address already in use

• InfluxDB Clientをインストールする

$ github.com/influxdata/influxdb/client/v2

⬛︎ まずは、CLIで基本動作を試してみる

実際の基本動作の確認は、こちらのGetting Startedページを参考にしました。

• まずは、CLIコマンドオプションを確認してみる

$ influx --help
Usage of influx:
  -version
       Display the version and exit.
  -host 'host name'
       Host to connect to.
  -port 'port #'
       Port to connect to.
  -database 'database name'
       Database to connect to the server.
  -password 'password'
      Password to connect to the server.  Leaving blank will prompt for password (--password '').
  -username 'username'
       Username to connect to the server.
  -ssl
        Use https for requests.
  -unsafeSsl
        Set this when connecting to the cluster using https and not use SSL verification.
  -execute 'command'
       Execute command and quit.
  -format 'json|csv|column'
       Format specifies the format of the server responses:  json, csv, or column.
  -precision 'rfc3339|h|m|s|ms|u|ns'
       Precision specifies the format of the timestamp:  rfc3339, h, m, s, ms, u or ns.
  -consistency 'any|one|quorum|all'
       Set write consistency level: any, one, quorum, or all
  -pretty
       Turns on pretty print for the json format.
  -import
       Import a previous database export from file
  -pps
       How many points per second the import will allow.  By default it is zero and will not throttle importing.
  -path
       Path to file to import
  -compressed
       Set to true if the import file is compressed

Examples:

    # Use influx in a non-interactive mode to query the database "metrics" and pretty print json:
    $ influx -database 'metrics' -execute 'select * from cpu' -format 'json' -pretty

    # Connect to a specific database on startup and set database context:
    $ influx -database 'metrics' -host 'localhost' -port '8086'

• 早速、DBを作成する

$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.13.0
InfluxDB shell version: 0.13.0
> CREATE DATABASE mydb
> SHOW DATABASES
name: databases
---------------
name
_internal
mydb

• mydbを使用可能な状態にする

> USE mydb
Using database mydb

• CLIで使用可能なコマンドを確認しておく

> help
Usage:
        connect <host:port>   connects to another node specified by host:port
        auth                  prompts for username and password
        pretty                toggles pretty print for the json format
        use <db_name>         sets current database
        format <format>       specifies the format of the server responses: json, csv, or column
        precision <format>    specifies the format of the timestamp: rfc3339, h, m, s, ms, u or ns
        consistency <level>   sets write consistency level: any, one, quorum, or all
        history               displays command history
        settings              outputs the current settings for the shell
        exit/quit/ctrl+d      quits the influx shell

        show databases        show database names
        show series           show series information
        show measurements     show measurement information
        show tag keys         show tag key information
        show field keys       show field key information

        A full list of influxql commands can be found at:
        https://docs.influxdata.com/influxdb/latest/query_language/spec/

• データを登録してみる

> INSERT cpu,host=serverA,region=us_west value=0.64

• 登録したデータ属性を確認してみる

> SHOW SERIES
key
cpu,host=serverA,region=us_west

• 登録したデータを参照してみる

> SELECT * FROM cpu
name: cpu
---------
time            host    region  value
1468794703195136385 serverA us_west 0.64

• ちなみに、InfluxDBのWeb-ui上からでも、CLI操作と同等なことが行えるようです。

⬛︎ golangから、時系列データを作成してみる

実際の基本動作の確認は、こちらのREADME.mdページを参考にしました。

(1)サンプルコードを準備する。

サンプリコードは、次のような動作になります。

• Create database --> DB"systemstats"を作成する
• Inserting Data --> ２０件の時系列データを作成する
• Querying Data --> 最初の１０件だけを表示する

sample_influxDB_client.go

package main 

import (
    "log"
    "time"
    "fmt"
    "math/rand"
    "github.com/influxdata/influxdb/client/v2"
)

const (
    MyDB = "systemstats"
    username = "bubba"
    password = "bumblebeetuna"
    MyMeasurement = "cpu_usage"
)

// queryDB convenience function to query the database
func queryDB(clnt client.Client, cmd string) (res []client.Result, err error) {
    q := client.Query{
        Command:  cmd,
        Database: MyDB,
    }
    if response, err := clnt.Query(q); err == nil {
        if response.Error() != nil {
            return res, response.Error()
        }
        res = response.Results
    } else {
        return res, err
    }
    return res, nil
}

func writePoints(clnt client.Client) {
    sampleSize := 20
    rand.Seed(42)

    bp, _ := client.NewBatchPoints(client.BatchPointsConfig{
        Database:  MyDB,
        Precision: "s",
    })

    for i := 0; i < sampleSize; i++ {
        time.Sleep(time.Second)
        regions := []string{"us-west1", "us-west2", "us-west3", "us-east1"}
        tags := map[string]string{
            "cpu":    "cpu-total",
            "host":   fmt.Sprintf("host%d", rand.Intn(1000)),
            "region": regions[rand.Intn(len(regions))],
        }

        idle := rand.Float64() * 100.0
        fields := map[string]interface{}{
            "idle": idle,
            "busy": 100.0 - idle,
        }

        pt, err := client.NewPoint(MyMeasurement, tags, fields, time.Now())
        if err != nil {
            log.Fatalln("Error: ", err)
        }
        bp.AddPoint(pt)
    }

    err := clnt.Write(bp)
    if err != nil {
        log.Fatal(err)
    }
}

func main() {
    // Make client
    clnt, err := client.NewHTTPClient(client.HTTPConfig{
        Addr: "http://localhost:8086",
        Username: username,
        Password: password,
    })

    if err != nil {
        log.Fatalln("Error: ", err)
    }

    // Create database
    _, err = queryDB(clnt, fmt.Sprintf("CREATE DATABASE %s", MyDB))
    if err != nil {
        log.Fatal(err)
    }

    // Inserting Data
    writePoints(clnt)

    // Querying Data
    q := fmt.Sprintf("SELECT * FROM %s LIMIT %d", MyMeasurement, 10)
    res, err := queryDB(clnt, q)
    if err != nil {
        log.Fatal(err)
    }

    for i, row := range res[0].Series[0].Values {
        t, err := time.Parse(time.RFC3339, row[0].(string))
        if err != nil {
            log.Fatal(err)
        }
        val := row[1]
        log.Printf("[%2d] %s: %s\n", i, t.Format(time.Stamp), val)
    }

}

(2)実際に、サンプルコードを動作させてみる。

• まず、DBアカウントを作成しておく

> create user bubba with password 'bumblebeetuna'
> grant all privileges to bubba

• 時系列データ作成用のサンプルコードを実行する

$ go run sample_influxDB_client.go 
2016/07/18 08:59:59 [ 0] Jul 17 23:59:40: 39.5906148441358
2016/07/18 08:59:59 [ 1] Jul 17 23:59:41: 61.680670007761435
2016/07/18 08:59:59 [ 2] Jul 17 23:59:42: 61.69553469502837
2016/07/18 08:59:59 [ 3] Jul 17 23:59:43: 78.20584254243772
2016/07/18 08:59:59 [ 4] Jul 17 23:59:44: 33.5989052315491
2016/07/18 08:59:59 [ 5] Jul 17 23:59:45: 28.40939195202982
2016/07/18 08:59:59 [ 6] Jul 17 23:59:46: 88.0668203236267
2016/07/18 08:59:59 [ 7] Jul 17 23:59:47: 77.28777698759987
2016/07/18 08:59:59 [ 8] Jul 17 23:59:48: 7.321156839368697
2016/07/18 08:59:59 [ 9] Jul 17 23:59:49: 78.89023835915611

• CLIから、登録し時系列データを参照してみる

$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.13.0
InfluxDB shell version: 0.13.0

> USE systemstats
Using database systemstats

> SELECT * FROM cpu_usage;
name: cpu_usage
---------------
time            busy            cpu     host    idle            region
1468799980000000000 39.5906148441358    cpu-total   host305 60.4093851558642    us-east1
1468799981000000000 61.680670007761435  cpu-total   host750 38.319329992238565  us-east1
1468799982000000000 61.69553469502837   cpu-total   host357 38.30446530497163   us-west1
1468799983000000000 78.20584254243772   cpu-total   host643 21.794157457562278  us-west2
1468799984000000000 33.5989052315491    cpu-total   host815 66.4010947684509    us-west1
1468799985000000000 28.40939195202982   cpu-total   host653 71.59060804797018   us-west1
1468799986000000000 88.0668203236267    cpu-total   host535 11.933179676373298  us-west1
1468799987000000000 77.28777698759987   cpu-total   host934 22.71222301240013   us-west1
1468799988000000000 7.321156839368697   cpu-total   host544 92.6788431606313    us-west2
1468799989000000000 78.89023835915611   cpu-total   host919 21.10976164084389   us-east1
1468799990000000000 10.738313757378009  cpu-total   host982 89.26168624262199   us-west3
1468799991000000000 2.6520038604342773  cpu-total   host610 97.34799613956572   us-west2
1468799992000000000 84.84195338561217   cpu-total   host974 15.15804661438783   us-east1
1468799993000000000 24.40410949424208   cpu-total   host507 75.59589050575792   us-east1
1468799994000000000 29.552453743568435  cpu-total   host824 70.44754625643156   us-west1
1468799995000000000 45.641703299549896  cpu-total   host512 54.358296700450104  us-west3
1468799996000000000 81.30658739342343   cpu-total   host429 18.693412606576576  us-east1
1468799997000000000 20.662632620528527  cpu-total   host392 79.33736737947147   us-west2
1468799998000000000 54.67951809721495   cpu-total   host72  45.32048190278505   us-west1
1468799999000000000 74.10616428036468   cpu-total   host401 25.893835719635323  us-west3

• 時系列データのtime値が、わかりにくいので、timestampのフォーマットを変更してみる

> precision rfc3339
> SELECT * FROM cpu_usage;
name: cpu_usage
---------------
time            busy            cpu     host    idle            region
2016-07-17T23:59:40Z    39.5906148441358    cpu-total   host305 60.4093851558642    us-east1
2016-07-17T23:59:41Z    61.680670007761435  cpu-total   host750 38.319329992238565  us-east1
2016-07-17T23:59:42Z    61.69553469502837   cpu-total   host357 38.30446530497163   us-west1
2016-07-17T23:59:43Z    78.20584254243772   cpu-total   host643 21.794157457562278  us-west2
2016-07-17T23:59:44Z    33.5989052315491    cpu-total   host815 66.4010947684509    us-west1
2016-07-17T23:59:45Z    28.40939195202982   cpu-total   host653 71.59060804797018   us-west1
2016-07-17T23:59:46Z    88.0668203236267    cpu-total   host535 11.933179676373298  us-west1
2016-07-17T23:59:47Z    77.28777698759987   cpu-total   host934 22.71222301240013   us-west1
2016-07-17T23:59:48Z    7.321156839368697   cpu-total   host544 92.6788431606313    us-west2
2016-07-17T23:59:49Z    78.89023835915611   cpu-total   host919 21.10976164084389   us-east1
2016-07-17T23:59:50Z    10.738313757378009  cpu-total   host982 89.26168624262199   us-west3
2016-07-17T23:59:51Z    2.6520038604342773  cpu-total   host610 97.34799613956572   us-west2
2016-07-17T23:59:52Z    84.84195338561217   cpu-total   host974 15.15804661438783   us-east1
2016-07-17T23:59:53Z    24.40410949424208   cpu-total   host507 75.59589050575792   us-east1
2016-07-17T23:59:54Z    29.552453743568435  cpu-total   host824 70.44754625643156   us-west1
2016-07-17T23:59:55Z    45.641703299549896  cpu-total   host512 54.358296700450104  us-west3
2016-07-17T23:59:56Z    81.30658739342343   cpu-total   host429 18.693412606576576  us-east1
2016-07-17T23:59:57Z    20.662632620528527  cpu-total   host392 79.33736737947147   us-west2
2016-07-17T23:59:58Z    54.67951809721495   cpu-total   host72  45.32048190278505   us-west1
2016-07-17T23:59:59Z    74.10616428036468   cpu-total   host401 25.893835719635323  us-west3

• InfluxDBのWeb-ui上から、時系列データを確認してみる

⬛︎ 最後に..

今回、InfluxDBの基本動作を試してみました。
事前に、スキーマ定義を行うことなく、いきなり、時系列データを登録できてしまう手軽さは、とても良い感じです。
あと、CLIの操作感も、とてもわかりやすかったですね。
さらに、Influxdbは、golangで書かれていますので、golangベースの外部アプリとの親和性がとても高いと感じました。