golangで、時系列データベース「InfluxDB」を試してみる

今回、時系列データベース”InfluxDB”の基本動作を試してみました。

⬛︎ InfluxDB環境構築

実際の構築にあたって、こちらのInstallationページを参考にしました。

(1) Linux環境準備

• Ubunts環境を確認する

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"

• golangをインストールする

$ vi $HOME/.profile

...(snip)
export GOPATH=$HOME/golang
export PATH=$GOPATH/bin:/usr/local/go/bin:$PATH

$ wget --no-check-certificate https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.6.2.linux-amd64.tar.gz
$ mkdir $HOME/golang
$ source .profile

$ go version
go version go1.6.2 linux/amd64

(2) InfluxDB環境構築

• InfluxDBをインストールする

$ sudo apt-get update
$ curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
$ echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

$ sudo apt-get update && sudo apt-get install influxdb

• InfluxDBを起動する

$ sudo service influxdb start
$ sudo service influxdb status
● influxdb.service - InfluxDB is an open-source, distributed, time series database
   Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2016-07-17 17:29:40 JST; 17s ago
     Docs: https://docs.influxdata.com/influxdb/
 Main PID: 1932 (sh)
    Tasks: 8 (limit: 512)
   Memory: 7.8M
      CPU: 53ms
   CGroup: /system.slice/influxdb.service
           ├─1932 /bin/sh -c /usr/bin/influxd -config /etc/influxdb/influxdb.conf  >>/dev/null 2>>/var/log/influxdb/influxd.log
           └─1934 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

Jul 17 17:29:40 influxdb systemd[1]: Started InfluxDB is an open-source, distributed, time series database.

• InfluxDBの動作を確認する

$ influxd

 8888888           .d888 888                   8888888b.  888888b.
   888            d88P"  888                   888  "Y88b 888  "88b
   888            888    888                   888    888 888  .88P
   888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
   888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
   888   888  888 888    888 888  888   X88K   888    888 888    888
   888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
 8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"

[run] 2016/07/17 17:32:11 InfluxDB starting, version 0.13.0, branch 0.13, commit e57fb88a051ee40fd9277094345fbd47bb4783ce
[run] 2016/07/17 17:32:11 Go version go1.6.2, GOMAXPROCS set to 1
[run] 2016/07/17 17:32:11 Using configuration at: /etc/influxdb/influxdb.conf
run: open server: listen: listen tcp :8088: bind: address already in use

• InfluxDB Clientをインストールする

$ github.com/influxdata/influxdb/client/v2

⬛︎ まずは、CLIで基本動作を試してみる

実際の基本動作の確認は、こちらのGetting Startedページを参考にしました。

• まずは、CLIコマンドオプションを確認してみる

$ influx --help
Usage of influx:
  -version
       Display the version and exit.
  -host 'host name'
       Host to connect to.
  -port 'port #'
       Port to connect to.
  -database 'database name'
       Database to connect to the server.
  -password 'password'
      Password to connect to the server.  Leaving blank will prompt for password (--password '').
  -username 'username'
       Username to connect to the server.
  -ssl
        Use https for requests.
  -unsafeSsl
        Set this when connecting to the cluster using https and not use SSL verification.
  -execute 'command'
       Execute command and quit.
  -format 'json|csv|column'
       Format specifies the format of the server responses:  json, csv, or column.
  -precision 'rfc3339|h|m|s|ms|u|ns'
       Precision specifies the format of the timestamp:  rfc3339, h, m, s, ms, u or ns.
  -consistency 'any|one|quorum|all'
       Set write consistency level: any, one, quorum, or all
  -pretty
       Turns on pretty print for the json format.
  -import
       Import a previous database export from file
  -pps
       How many points per second the import will allow.  By default it is zero and will not throttle importing.
  -path
       Path to file to import
  -compressed
       Set to true if the import file is compressed

Examples:

    # Use influx in a non-interactive mode to query the database "metrics" and pretty print json:
    $ influx -database 'metrics' -execute 'select * from cpu' -format 'json' -pretty

    # Connect to a specific database on startup and set database context:
    $ influx -database 'metrics' -host 'localhost' -port '8086'

• 早速、DBを作成する

$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.13.0
InfluxDB shell version: 0.13.0
> CREATE DATABASE mydb
> SHOW DATABASES
name: databases
---------------
name
_internal
mydb

• mydbを使用可能な状態にする

> USE mydb
Using database mydb

• CLIで使用可能なコマンドを確認しておく

> help
Usage:
        connect <host:port>   connects to another node specified by host:port
        auth                  prompts for username and password
        pretty                toggles pretty print for the json format
        use <db_name>         sets current database
        format <format>       specifies the format of the server responses: json, csv, or column
        precision <format>    specifies the format of the timestamp: rfc3339, h, m, s, ms, u or ns
        consistency <level>   sets write consistency level: any, one, quorum, or all
        history               displays command history
        settings              outputs the current settings for the shell
        exit/quit/ctrl+d      quits the influx shell

        show databases        show database names
        show series           show series information
        show measurements     show measurement information
        show tag keys         show tag key information
        show field keys       show field key information

        A full list of influxql commands can be found at:
        https://docs.influxdata.com/influxdb/latest/query_language/spec/

• データを登録してみる

> INSERT cpu,host=serverA,region=us_west value=0.64

• 登録したデータ属性を確認してみる

> SHOW SERIES
key
cpu,host=serverA,region=us_west

• 登録したデータを参照してみる

> SELECT * FROM cpu
name: cpu
---------
time			host	region	value
1468794703195136385	serverA	us_west	0.64

• ちなみに、InfluxDBのWeb-ui上からでも、CLI操作と同等なことが行えるようです。

⬛︎ golangから、時系列データを作成してみる

実際の基本動作の確認は、こちらのREADME.mdページを参考にしました。

(1)サンプルコードを準備する。

サンプリコードは、次のような動作になります。

• Create database --> DB"systemstats"を作成する
• Inserting Data --> ２０件の時系列データを作成する
• Querying Data --> 最初の１０件だけを表示する

sample_influxDB_client.go

package main 

import (
    "log"
    "time"
    "fmt"
    "math/rand"
    "github.com/influxdata/influxdb/client/v2"
)

const (
    MyDB = "systemstats"
    username = "bubba"
    password = "bumblebeetuna"
    MyMeasurement = "cpu_usage"
)

// queryDB convenience function to query the database
func queryDB(clnt client.Client, cmd string) (res []client.Result, err error) {
    q := client.Query{
        Command:  cmd,
        Database: MyDB,
    }
    if response, err := clnt.Query(q); err == nil {
        if response.Error() != nil {
            return res, response.Error()
        }
        res = response.Results
    } else {
        return res, err
    }
    return res, nil
}

func writePoints(clnt client.Client) {
    sampleSize := 20
    rand.Seed(42)

    bp, _ := client.NewBatchPoints(client.BatchPointsConfig{
        Database:  MyDB,
        Precision: "s",
    })

    for i := 0; i < sampleSize; i++ {
        time.Sleep(time.Second)
        regions := []string{"us-west1", "us-west2", "us-west3", "us-east1"}
        tags := map[string]string{
            "cpu":    "cpu-total",
            "host":   fmt.Sprintf("host%d", rand.Intn(1000)),
            "region": regions[rand.Intn(len(regions))],
        }

        idle := rand.Float64() * 100.0
        fields := map[string]interface{}{
            "idle": idle,
            "busy": 100.0 - idle,
        }

        pt, err := client.NewPoint(MyMeasurement, tags, fields, time.Now())
        if err != nil {
            log.Fatalln("Error: ", err)
        }
        bp.AddPoint(pt)
    }

    err := clnt.Write(bp)
    if err != nil {
        log.Fatal(err)
    }
}

func main() {
    // Make client
    clnt, err := client.NewHTTPClient(client.HTTPConfig{
        Addr: "http://localhost:8086",
        Username: username,
        Password: password,
    })

    if err != nil {
        log.Fatalln("Error: ", err)
    }

    // Create database
    _, err = queryDB(clnt, fmt.Sprintf("CREATE DATABASE %s", MyDB))
    if err != nil {
        log.Fatal(err)
    }

    // Inserting Data
    writePoints(clnt)

    // Querying Data
    q := fmt.Sprintf("SELECT * FROM %s LIMIT %d", MyMeasurement, 10)
    res, err := queryDB(clnt, q)
    if err != nil {
        log.Fatal(err)
    }

    for i, row := range res[0].Series[0].Values {
        t, err := time.Parse(time.RFC3339, row[0].(string))
        if err != nil {
            log.Fatal(err)
        }
        val := row[1]
        log.Printf("[%2d] %s: %s\n", i, t.Format(time.Stamp), val)
    }

}

(2)実際に、サンプルコードを動作させてみる。

• まず、DBアカウントを作成しておく

> create user bubba with password 'bumblebeetuna'
> grant all privileges to bubba

• 時系列データ作成用のサンプルコードを実行する

$ go run sample_influxDB_client.go 
2016/07/18 08:59:59 [ 0] Jul 17 23:59:40: 39.5906148441358
2016/07/18 08:59:59 [ 1] Jul 17 23:59:41: 61.680670007761435
2016/07/18 08:59:59 [ 2] Jul 17 23:59:42: 61.69553469502837
2016/07/18 08:59:59 [ 3] Jul 17 23:59:43: 78.20584254243772
2016/07/18 08:59:59 [ 4] Jul 17 23:59:44: 33.5989052315491
2016/07/18 08:59:59 [ 5] Jul 17 23:59:45: 28.40939195202982
2016/07/18 08:59:59 [ 6] Jul 17 23:59:46: 88.0668203236267
2016/07/18 08:59:59 [ 7] Jul 17 23:59:47: 77.28777698759987
2016/07/18 08:59:59 [ 8] Jul 17 23:59:48: 7.321156839368697
2016/07/18 08:59:59 [ 9] Jul 17 23:59:49: 78.89023835915611

• CLIから、登録し時系列データを参照してみる

$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.13.0
InfluxDB shell version: 0.13.0

> USE systemstats
Using database systemstats

> SELECT * FROM cpu_usage;
name: cpu_usage
---------------
time			busy			cpu		host	idle			region
1468799980000000000	39.5906148441358	cpu-total	host305	60.4093851558642	us-east1
1468799981000000000	61.680670007761435	cpu-total	host750	38.319329992238565	us-east1
1468799982000000000	61.69553469502837	cpu-total	host357	38.30446530497163	us-west1
1468799983000000000	78.20584254243772	cpu-total	host643	21.794157457562278	us-west2
1468799984000000000	33.5989052315491	cpu-total	host815	66.4010947684509	us-west1
1468799985000000000	28.40939195202982	cpu-total	host653	71.59060804797018	us-west1
1468799986000000000	88.0668203236267	cpu-total	host535	11.933179676373298	us-west1
1468799987000000000	77.28777698759987	cpu-total	host934	22.71222301240013	us-west1
1468799988000000000	7.321156839368697	cpu-total	host544	92.6788431606313	us-west2
1468799989000000000	78.89023835915611	cpu-total	host919	21.10976164084389	us-east1
1468799990000000000	10.738313757378009	cpu-total	host982	89.26168624262199	us-west3
1468799991000000000	2.6520038604342773	cpu-total	host610	97.34799613956572	us-west2
1468799992000000000	84.84195338561217	cpu-total	host974	15.15804661438783	us-east1
1468799993000000000	24.40410949424208	cpu-total	host507	75.59589050575792	us-east1
1468799994000000000	29.552453743568435	cpu-total	host824	70.44754625643156	us-west1
1468799995000000000	45.641703299549896	cpu-total	host512	54.358296700450104	us-west3
1468799996000000000	81.30658739342343	cpu-total	host429	18.693412606576576	us-east1
1468799997000000000	20.662632620528527	cpu-total	host392	79.33736737947147	us-west2
1468799998000000000	54.67951809721495	cpu-total	host72	45.32048190278505	us-west1
1468799999000000000	74.10616428036468	cpu-total	host401	25.893835719635323	us-west3

• 時系列データのtime値が、わかりにくいので、timestampのフォーマットを変更してみる

> precision rfc3339
> SELECT * FROM cpu_usage;
name: cpu_usage
---------------
time			busy			cpu		host	idle			region
2016-07-17T23:59:40Z	39.5906148441358	cpu-total	host305	60.4093851558642	us-east1
2016-07-17T23:59:41Z	61.680670007761435	cpu-total	host750	38.319329992238565	us-east1
2016-07-17T23:59:42Z	61.69553469502837	cpu-total	host357	38.30446530497163	us-west1
2016-07-17T23:59:43Z	78.20584254243772	cpu-total	host643	21.794157457562278	us-west2
2016-07-17T23:59:44Z	33.5989052315491	cpu-total	host815	66.4010947684509	us-west1
2016-07-17T23:59:45Z	28.40939195202982	cpu-total	host653	71.59060804797018	us-west1
2016-07-17T23:59:46Z	88.0668203236267	cpu-total	host535	11.933179676373298	us-west1
2016-07-17T23:59:47Z	77.28777698759987	cpu-total	host934	22.71222301240013	us-west1
2016-07-17T23:59:48Z	7.321156839368697	cpu-total	host544	92.6788431606313	us-west2
2016-07-17T23:59:49Z	78.89023835915611	cpu-total	host919	21.10976164084389	us-east1
2016-07-17T23:59:50Z	10.738313757378009	cpu-total	host982	89.26168624262199	us-west3
2016-07-17T23:59:51Z	2.6520038604342773	cpu-total	host610	97.34799613956572	us-west2
2016-07-17T23:59:52Z	84.84195338561217	cpu-total	host974	15.15804661438783	us-east1
2016-07-17T23:59:53Z	24.40410949424208	cpu-total	host507	75.59589050575792	us-east1
2016-07-17T23:59:54Z	29.552453743568435	cpu-total	host824	70.44754625643156	us-west1
2016-07-17T23:59:55Z	45.641703299549896	cpu-total	host512	54.358296700450104	us-west3
2016-07-17T23:59:56Z	81.30658739342343	cpu-total	host429	18.693412606576576	us-east1
2016-07-17T23:59:57Z	20.662632620528527	cpu-total	host392	79.33736737947147	us-west2
2016-07-17T23:59:58Z	54.67951809721495	cpu-total	host72	45.32048190278505	us-west1
2016-07-17T23:59:59Z	74.10616428036468	cpu-total	host401	25.893835719635323	us-west3

• InfluxDBのWeb-ui上から、時系列データを確認してみる

⬛︎ 最後に..

今回、InfluxDBの基本動作を試してみました。
事前に、スキーマ定義を行うことなく、いきなり、時系列データを登録できてしまう手軽さは、とても良い感じです。
あと、CLIの操作感も、とてもわかりやすかったですね。
さらに、Influxdbは、golangで書かれていますので、golangベースの外部アプリとの親和性がとても高いと感じました。