はじめに
Redashとは、データソースに接続・クエリ結果の抽出ができ、目的に沿ってデータやクエリ結果をダッシュボード上で自由に可視化することができるBIツール。
データソースは広くサポートしている。
AWS周りでは、下記サービスがサポートされている。
- Amazon Athena
- Amazon CloudWatch
- Amazon CloudWatch Logs Insights
- Amazon DynamoDB
- Amazon Redshift
アーキテクチャ
下記コンポーネントを使い構築する(コード管理はterraformを使用)
- Redash (Web)Server/Worker -> ECS on Fargate (image: redash/redash:10.1.0.b50633 )
- Redis -> ECS on Fargate (iamge: redis:3-alpine)
- PostgreSQL -> RDS
- Data Source -> Athena
構築手順
SecurityGroup
- ELB: Redash使用予定端末のセグメントからの接続(HTTP:80)のみ許可
(今回とりあえずということでCertificationの設定を省いたが、最終的にはHTTPSで接続させる) - ECS on Fargate: ELBのSecurityGroupIDからの接続(TCP:5000)のみ許可
- RDS: ECS on Fargate-ServiceのSecurityGroupからの接続(TCP:5432)のみ許可
RDSを構築
1. インスタンスを作成
resource "aws_db_instance" "redash-test-rds" {
allocated_storage = 20
availability_zone = "us-east-1a"
engine = "postgres"
engine_version = "13.7"
instance_class = "db.t3.micro"
username = "postgres"
username = "postgres"
skip_final_snapshot = true
identifier = "redash-test-rds"
port = 5432
vpc_security_group_ids = [
"sg-XXXXX",
]
db_subnet_group_name = "default"
}
2. redash user作成
- 初期設定として、redash userを作成しておく
psql -h test-redash-db.XXXXX.us-east-1.rds.amazonaws.com -U postgres -p 5432
Password for user postgres: *****
postgres=> CREATE USER redash WITH PASSWORD redash123 CREATEDB; # username=redash, password=redash123で設定する場合
3. redash database作成
- 初期設定として、redash databaseを作成しておく
postgres=> CREATE DATABASE redash;
postgres=> GRANT ALL ON ALL TABLES IN SCHEMA public TO redash;
Secrets Managerを構築
- 環境変数[REDASH_COOKIE_SECRET, REDASH_SECRET_KEY, REDASH_DATABASE_URL]は機密情報を含むため、予めSecretを作成しておく
1. Secretを作成
resource "aws_secretsmanager_secret" "redash-test-secretmanager" {
name = "redash-test-secret"
}
2. Secretの値を作成
参照するSecretの値として下記3つを登録
REDASH_COOKIE_SECRET
Redash起動に伴い、ユーザ認証、Cookie署名、セッション情報保存などを暗号化する際に必要になる。
ランダム値を設定する必要があり、下記いずれかのコマンドによりキーの値を作成する。
$ python -c 'import secrets; print(secrets.token_hex())'
$ pwgen -1s 64
REDASH_SECRET_KEY
Redash起動に伴い、各種設定を暗号化する際に必要になる。
明示的に設定しない場合、REDASH_COOKIE_SECRETを代わりに使用する仕様にはなっているが、セキュリティを確保するために、REDASH_SECRET_KEYも設定することが推奨されている。
REDASH_COOKIE_SECRET同様のやり方で、ランダム値を設定する必要がある。
REDASH_DATABASE_URL
下記の文法でRDSのURLを設定する
postgresql://${PostgresUsername}:${PostgresPassword}@postgres:5432/${PostgresDBName}
ex)
・ユーザ名: redash
・パスワード: redash123
・DB名: redash
-> postgresql://redash:redash123@postgres:5432/redash
参考-> Environment Variables Settings
ALBを構築
- TargetGroupのリスナーに設定には、Redash Web ServerがListenしているHTTP/5000を設定 ★1
- health check用のAPIエンドポイントとして用意されている/ping を使う ★2
参考-> Developer Installation Guide
resource "aws_lb" "redash-test-alb" {
name = "redash-test-ELB"
internal = false
load_balancer_type = "application"
security_groups = [
"sg-XXXXX",
]
subnets = [
"subnet-XXXXX",
"subnet-XXXXX",
"subnet-XXXXX",
]
}
resource "aws_lb_target_group" "redash-test-tg" {
name = "redash-test-TG"
port = 5000 # ★1
protocol = "HTTP" # ★1
target_type = "ip" # ★1
vpc_id = "vpc-XXXXX"
health_check {
enabled = true
healthy_threshold = 5
interval = 30
path = "/ping" # ★2
port = "traffic-port" # ★2
protocol = "HTTP" # ★2
timeout = 5
unhealthy_threshold = 2
}
}
resource "aws_lb_listener" "redash-test-listener" {
load_balancer_arn = "arn:aws:elasticloadbalancing:us-east-1:XXXXX:loadbalancer/app/redash-test-ELB/XXXXX"
port = 80
protocol = "HTTP"
default_action {
order = 1
target_group_arn = "arn:aws:elasticloadbalancing:us-east-1:XXXXX:targetgroup/redash-test-TG/XXXXX"
type = "forward"
}
}
ECS on Fargateを構築
1. Serviceと紐づけるTask定義を作成
- Redash Web Server, Redash Worker, Redis の3つのContainer分を作成する
- nameをContainer毎に設定(任意の値でOK) ★1
- commandをContainer毎に適切に設定 (Redashの/app/bin/docker-entrypointファイルに記載の関数名=command名になる ) ★2
- docker-compose.ymlを参考に、Redash Web ServerとRedisはportMappingsを設定する ★3
- Redisのimageはdocker-compose.ymlに記載のimage名を設定 ★4
- Developer Installation Guideを参考に、SecretManagerで設定したもの+PYTHONUNBUFFERED, PYTHONUNBUFFEREDの環境変数を設定
resource "aws_ecs_task_definition" "redash-test-ecs-task-definition" {
family = "redash-test-Task"
cpu = "4096"
execution_role_arn = "arn:aws:iam::XXXXX:role/ecsTaskExecutionRole"
memory = "16384"
network_mode = "awsvpc"
requires_compatibilities = [
"FARGATE",
]
runtime_platform {
operating_system_family = "LINUX"
}
container_definitions = jsonencode(
[
### 以下RedisContainerのTask定義 ###
{
name = "redis" #★1
image = "redis:3-alpine" #★4
cpu = 0
essential = true
portMappings = [
{
containerPort = 6379 #★3
hostPort = 6379 #★3
protocol = "tcp" #★3
},
]
environment = [
{
name = "PYTHONUNBUFFERED"
value = "0"
},
{
name = "REDASH_LOG_LEVEL"
value = "INFO"
},
]
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = "/ecs/redash-test-Task"
awslogs-region = "us-east-1"
awslogs-stream-prefix = "ecs"
}
}
secrets = [
{
name = "REDASH_COOKIE_SECRET"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_COOKIE_SECRET::"
},
{
name = "REDASH_DATABASE_URL"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_DATABASE_URL::"
},
{
name = "REDASH_SECRET_KEY"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_SECRET_KEY::"
},
]
},
### 以下Redash Web ServerのTask定義 ###
{
name = "server" #★1
image = "redash/redash:10.1.0.b50633"
cpu = 0
#essential = true
portMappings = [
{
containerPort = 5000 #★3
hostPort = 5000 #★3
protocol = "tcp" #★3
},
]
command = [
"server", #★2
]
environment = [
{
name = "PYTHONUNBUFFERED"
value = "0"
},
{
name = "REDASH_LOG_LEVEL"
value = "INFO"
},
]
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = "/ecs/redash-test-Task"
awslogs-region = "us-east-1"
awslogs-stream-prefix = "ecs"
}
}
secrets = [
{
name = "REDASH_COOKIE_SECRET"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_COOKIE_SECRET::"
},
{
name = "REDASH_DATABASE_URL"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_DATABASE_URL::"
},
{
name = "REDASH_SECRET_KEY"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_SECRET_KEY::" },
]
},
### 以下Redash Workerのタスク定義 ###
{
name = "worker" #★1
image = "redash/redash:10.1.0.b50633"
cpu = 0
essential = true
portMappings = []
command = [
"worker", #★2
]
environment = [
{
name = "PYTHONUNBUFFERED"
value = "0"
},
{
name = "REDASH_LOG_LEVEL"
value = "INFO"
},
]
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = "/ecs/redash-test-Task"
awslogs-region = "us-east-1"
awslogs-stream-prefix = "ecs"
}
}
secrets = [
{
name = "REDASH_COOKIE_SECRET"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_COOKIE_SECRET::"
},
{
name = "REDASH_DATABASE_URL"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_DATABASE_URL::"
},
{
name = "REDASH_SECRET_KEY"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_SECRET_KEY::"
},
]
},
]
)
}
2. 初期設定用のTask定義(createdb)を作成
- Redash Web Server、create_dbの2つのContainer分を作成する
- nameをContainer毎に設定(任意の値でOK)
- commandをContainer毎に適切に設定 (Redashの/app/bin/docker-entrypointファイルに記載の関数名=command名になる )
- docker-compose.ymlを参考に、Redash Web ServerはportMappingsを設定する
resource "aws_ecs_task_definition" "redash-test-ecs-task-definition-createdb" {
family = "redash-test-Task-createdb"
cpu = "4096"
execution_role_arn = "arn:aws:iam::XXXXX:role/ecsTaskExecutionRole"
memory = "16384"
network_mode = "awsvpc"
requires_compatibilities = [
"FARGATE",
]
runtime_platform {
operating_system_family = "LINUX"
}
container_definitions = jsonencode(
[
### 以下Redash Web ServerのTask定義 ###
{
name = "server" #★1
image = "redash/redash:10.1.0.b50633"
cpu = 0
#essential = true
portMappings = [
{
containerPort = 5000 #★3
hostPort = 5000 #★3
protocol = "tcp" #★3
},
]
command = [
"server", #★2
]
environment = [
{
name = "PYTHONUNBUFFERED"
value = "0"
},
{
name = "REDASH_LOG_LEVEL"
value = "INFO"
},
]
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = "/ecs/redash-test-Task"
awslogs-region = "us-east-1"
awslogs-stream-prefix = "ecs"
}
}
secrets = [
{
name = "REDASH_COOKIE_SECRET"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_COOKIE_SECRET::"
},
{
name = "REDASH_DATABASE_URL"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_DATABASE_URL::"
},
{
name = "REDASH_SECRET_KEY"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_SECRET_KEY::" },
]
},
### 以下Redash Createdbのタスク定義 ###
{
name = "create_db" #★1
image = "redash/redash:10.1.0.b50633"
cpu = 0
essential = true
portMappings = []
command = [
"create_db", #★2
]
environment = [
{
name = "PYTHONUNBUFFERED"
value = "0"
},
{
name = "REDASH_LOG_LEVEL"
value = "INFO"
},
]
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = "/ecs/redash-test-Task"
awslogs-region = "us-east-1"
awslogs-stream-prefix = "ecs"
}
}
secrets = [
{
name = "REDASH_COOKIE_SECRET"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_COOKIE_SECRET::"
},
{
name = "REDASH_DATABASE_URL"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_DATABASE_URL::"
},
{
name = "REDASH_SECRET_KEY"
valueFrom = "arn:aws:secretsmanager:us-east-1:XXXXX:secret:redash-test-secret-XXXX:REDASH_SECRET_KEY::"
},
]
},
]
)
}
3. ClusterとServiceを作成
resource "aws_ecs_cluster" "redash-test-ecs-cluster" {
capacity_providers = [
"FARGATE",
"FARGATE_SPOT",
]
name = "redash-test-ecs-ECSCluster"
setting {
name = "containerInsights"
value = "disabled"
}
}
resource "aws_ecs_service" "redash-test-ecs-service" {
name = "redash-test-ecs-ECSService"
cluster = "arn:aws:ecs:us-east-1:XXXXX:cluster/redash-test-ecs-ECSCluster"
desired_count = 1
enable_ecs_managed_tags = true
iam_role = "AWSServiceRoleForECS"
launch_type = "FARGATE"
task_definition = "redash-test-Task:1"
load_balancer {
container_name = "server"
container_port = 5000
target_group_arn = "arn:aws:elasticloadbalancing:us-east-1:XXXXX:targetgroup/redash-test-TG/XXXXX"
}
network_configuration {
assign_public_ip = true
security_groups = [
"sg-XXXXX",
]
subnets = [
"subnet-XXXXX",
]
}
}
Container起動
1. 初期設定のため、redash-test-Task-createdbを起動
- 最初の1回だけ、初期設定のために手動実行する。
- コンテナが起動し、数分で終了する。
- CloudWatch ロググループからエラーが出ていないことを確認する。
2. redash-test-ecs-serviceを起動
- CloudWatch ロググループからエラーが出ていないことを確認する。
- http://[ELBのDNS名] にアクセスし、RedashのSetUP画面が出ることを確認する
おわりに
初のECS on Fargateということもあり、いろいろと躓いたので、お役に立てたらうれしいです。