やりたいこと
社内向けに DataFactory のハンズオンを開催した際に、環境を丸ごと構築・削除できるように Terraform でやってみました。その際の手順などをまとめます。
同じようなことをやろうしている方の参考になれば幸いです！
作成するパイプライン
以下のようなパイプラインを Terraform で構築する。構成としてはメダリオンアーキテクチャを採用して、Raw -> Bronze -> Silver -> Gold という流れでデータをステージングしていく。
ソースとなるデータは2つ (CSV) で、最終的にその2つを JOIN して Gold データを作成する。また、ファイル名やストレージアカウントのコンテナ名はパイプラインのパラメータとして設定している。
Terraform コード
前提
- リソースグループは作成済みとする
- 既存リソースグループに対して新規のリソースを作成していく
- ADF からストレージアカウントへのアクセスには、システム割当マネージド ID を使う
プロバイダ設定
Terraform コード内で使う各プロバイダの設定。
provider.tf
provider.tf
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = ">=3.64.0"
}
azuread = {
source = "hashicorp/azuread"
version = ">=2.39.0"
}
time = {
source = "hashicorp/time"
version = ">=0.9.1"
}
local = {
source = "hashicorp/local"
version = ">=2.4.0"
}
}
}
provider "azurerm" {
features {}
}
パラメータ設定
コード内で設定するパラメータなどを local.tf でローカル変数として設定する。
local.tf
local.tf
locals {
rg_name = "YOUR_RESOURCE_GROUP_NAME"
suffix = "YOUR_SUFFIX"
default_tags = {
Description = "ADF Handson"
Note = "This resource is created by Terraform"
}
storage_account = {
names = {
raw = join("", ["st", replace(local.suffix, "-", ""), "raw"]),
bronze = join("", ["st", replace(local.suffix, "-", ""), "bronze"]),
silver = join("", ["st", replace(local.suffix, "-", ""), "silver"]),
gold = join("", ["st", replace(local.suffix, "-", ""), "gold"])
}
account_tier = "Standard"
account_replication_type = "LRS"
}
container = {
names = {
sample = "sample"
}
}
blob = {
sources = {
sample1 = {
name = "sample1.csv"
source = "data/sample1.csv"
}
sample2 = {
name = "sample2.csv"
source = "data/sample2.csv"
}
}
}
data_factory = {
name = "adf-${local.suffix}"
}
}
既存リソースの読み込み
data として既存のリソースなどを読み込む。今回は既存のリソースグループを読み込む。また、カレントユーザの情報を読み込む (サブスクリプション ID の取得用)。
data.tf
data.tf
data "azurerm_resource_group" "my_rg" {
name = local.rg_name
}
data "azurerm_client_config" "current" {}
Blob データ
パイプラインで処理するデータも Terraform コードであらかじめストレージアカウントへアップロードしておく。そのためのサンプルデータ。data/ 下に保存する。
sample1.csv
sample1.csv
sample1_id,column11,column12,column13,column14,column15,useless_column
0,101,102,103,104,105,hoge
1,111,112,113,114,115,hoge
2,121,122,123,124,125,hoge
3,131,132,133,134,135,hoge
4,141,142,143,144,145,hoge
5,151,152,153,154,155,hoge
6,161,162,163,164,165,hoge
7,171,172,173,174,175,hoge
8,181,182,183,184,185,hoge
9,191,192,193,194,195,hoge
sample2.csv
sample2.csv
sample2_id,column21,column22,column23,column24,column25
0,201,202,203,204,205
1,211,212,213,214,215
2,221,222,223,224,225
3,231,232,233,234,235
4,241,242,243,244,245
5,251,252,253,254,255
6,261,262,263,264,265
7,271,272,273,274,275
8,281,272,283,284,285
9,291,292,293,294,295
ストレージアカウントの作成
各ステージ (Raw ～ Gold) のデータを格納するためのストレージアカウントとコンテナを作成する。同時に、Raw には Blob データをローカルファイルからアップロードしておく。
create_storage_account.tf
create_storage_account.tf
resource "azurerm_storage_account" "storages" {
for_each = local.storage_account.names
name = each.value
resource_group_name = data.azurerm_resource_group.my_rg.name
location = data.azurerm_resource_group.my_rg.location
account_tier = local.storage_account.account_tier
account_replication_type = local.storage_account.account_replication_type
is_hns_enabled = true
tags = local.default_tags
}
resource "time_sleep" "wait_storage_account_created" {
depends_on = [azurerm_storage_account.storages]
create_duration = "60s"
}
resource "azurerm_storage_container" "st_containers" {
depends_on = [time_sleep.wait_storage_account_created] # wait for storage accounts creation
for_each = {
for item in setproduct(
[for st_name in local.storage_account.names : st_name],
[for container_name in local.container.names : container_name]
) : join("-", item) => item
}
storage_account_name = each.value[0]
container_access_type = "container"
name = each.value[1]
}
resource "time_sleep" "wait_container_created" {
depends_on = [azurerm_storage_container.st_containers]
create_duration = "60s"
}
resource "azurerm_storage_blob" "st_blob_raw" {
depends_on = [time_sleep.wait_container_created] # wait for containers creation
for_each = local.blob.sources
name = each.value.name
storage_account_name = azurerm_storage_account.storages["raw"].name
storage_container_name = azurerm_storage_container.st_containers["${local.storage_account.names["raw"]}-${local.container.names["sample"]}"].name
type = "Block"
source = each.value.source
}
Dataflow 定義
Gold データを作成する Dataflow の定義。パイプライン作成時にこの定義を利用する。pipeline_definitions/ に保存する。
create_gold_data_sample.txt
create_gold_data_sample.txt
source(output(
sample1_id as short,
column11 as short,
column12 as short,
column13 as short,
column14 as short,
column15 as short
),
allowSchemaDrift: true,
validateSchema: false,
ignoreNoFilesFound: false) ~> sourceSilverSample1
source(output(
sample2_id as short,
column21 as short,
column22 as short,
column23 as short,
column24 as short,
column25 as short
),
allowSchemaDrift: true,
validateSchema: false,
ignoreNoFilesFound: false) ~> sourceSilverSample2
sourceSilverSample1, sourceSilverSample2 join(sample1_id == sample2_id,
joinType:'inner',
matchType:'exact',
ignoreSpaces: false,
broadcast: 'auto') ~> joinSample1Sample2
joinSample1Sample2 sink(allowSchemaDrift: true,
validateSchema: false,
filePattern:'sample.csv',
umask: 0022,
preCommands: [],
postCommands: [],
skipDuplicateMapInputs: true,
skipDuplicateMapOutputs: true,
mapColumn(
sample_id = sample1_id,
column11,
column12,
column13,
column14,
column15,
column21,
column22,
column23,
column24,
column25
)) ~> sinkGold
パイプライン定義
作成するパイプライン全体の定義。pipeline_definitions/ に保存する。
sample.json
sample.json
[
{
"dependsOn": [],
"name": "CopyToBronzeSample1",
"inputs": [
{
"parameters": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_1"
}
},
"referenceName": "DelimitedTextRaw",
"type": "DatasetReference"
}
],
"outputs": [
{
"parameters": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_1"
}
},
"referenceName": "DelimitedTextBronze",
"type": "DatasetReference"
}
],
"policy": {
"retry": 0,
"retryIntervalInSeconds": 30,
"secureInput": false,
"secureOutput": false,
"timeout": "0.12:00:00"
},
"type": "Copy",
"typeProperties": {
"enableStaging": false,
"sink": {
"formatSettings": {
"fileExtension": ".txt",
"quoteAllText": true,
"type": "DelimitedTextWriteSettings"
},
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"type": "DelimitedTextSink"
},
"source": {
"formatSettings": {
"compressionProperties": null,
"type": "DelimitedTextReadSettings"
},
"storeSettings": {
"enablePartitionDiscovery": false,
"recursive": true,
"type": "AzureBlobFSReadSettings"
},
"type": "DelimitedTextSource"
},
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
},
"mappings": [
{
"sink": {
"name": "sample1_id",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "sample1_id",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column11",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column11",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column12",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column12",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column13",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column13",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column14",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column14",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column15",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column15",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "useless_column",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "useless_column",
"physicalType": "String",
"type": "String"
}
}
]
}
},
"userProperties": []
},
{
"dependsOn": [
{
"activity": "CopyToBronzeSample1",
"dependencyConditions": [
"Succeeded"
]
}
],
"name": "CopyToSilverSample1",
"inputs": [
{
"parameters": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_1"
}
},
"referenceName": "DelimitedTextBronze",
"type": "DatasetReference"
}
],
"outputs": [
{
"parameters": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_1"
}
},
"referenceName": "DelimitedTextSilver",
"type": "DatasetReference"
}
],
"policy": {
"retry": 0,
"retryIntervalInSeconds": 30,
"secureInput": false,
"secureOutput": false,
"timeout": "0.12:00:00"
},
"type": "Copy",
"typeProperties": {
"enableStaging": false,
"sink": {
"formatSettings": {
"fileExtension": ".txt",
"quoteAllText": true,
"type": "DelimitedTextWriteSettings"
},
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"type": "DelimitedTextSink"
},
"source": {
"formatSettings": {
"compressionProperties": null,
"type": "DelimitedTextReadSettings"
},
"storeSettings": {
"enablePartitionDiscovery": false,
"recursive": true,
"type": "AzureBlobFSReadSettings"
},
"type": "DelimitedTextSource"
},
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
},
"mappings": [
{
"sink": {
"name": "sample1_id",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "sample1_id",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column11",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column11",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column12",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column12",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column13",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column13",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column14",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column14",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column15",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column15",
"physicalType": "String",
"type": "String"
}
}
]
}
},
"userProperties": []
},
{
"dependsOn": [],
"name": "CopyToBronzeSample2",
"inputs": [
{
"parameters": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_2"
}
},
"referenceName": "DelimitedTextRaw",
"type": "DatasetReference"
}
],
"outputs": [
{
"parameters": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_2"
}
},
"referenceName": "DelimitedTextBronze",
"type": "DatasetReference"
}
],
"policy": {
"retry": 0,
"retryIntervalInSeconds": 30,
"secureInput": false,
"secureOutput": false,
"timeout": "0.12:00:00"
},
"type": "Copy",
"typeProperties": {
"enableStaging": false,
"sink": {
"formatSettings": {
"fileExtension": ".txt",
"quoteAllText": true,
"type": "DelimitedTextWriteSettings"
},
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"type": "DelimitedTextSink"
},
"source": {
"formatSettings": {
"compressionProperties": null,
"type": "DelimitedTextReadSettings"
},
"storeSettings": {
"enablePartitionDiscovery": false,
"recursive": true,
"type": "AzureBlobFSReadSettings"
},
"type": "DelimitedTextSource"
},
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
},
"mappings": [
{
"sink": {
"name": "sample2_id",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "sample2_id",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column21",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column21",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column22",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column22",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column23",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column23",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column24",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column24",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column25",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column25",
"physicalType": "String",
"type": "String"
}
}
]
}
},
"userProperties": []
},
{
"dependsOn": [
{
"activity": "CopyToBronzeSample2",
"dependencyConditions": [
"Succeeded"
]
}
],
"name": "CopyToSilverSample2",
"inputs": [
{
"parameters": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_2"
}
},
"referenceName": "DelimitedTextBronze",
"type": "DatasetReference"
}
],
"outputs": [
{
"parameters": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_2"
}
},
"referenceName": "DelimitedTextSilver",
"type": "DatasetReference"
}
],
"policy": {
"retry": 0,
"retryIntervalInSeconds": 30,
"secureInput": false,
"secureOutput": false,
"timeout": "0.12:00:00"
},
"type": "Copy",
"typeProperties": {
"enableStaging": false,
"sink": {
"formatSettings": {
"fileExtension": ".txt",
"quoteAllText": true,
"type": "DelimitedTextWriteSettings"
},
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"type": "DelimitedTextSink"
},
"source": {
"formatSettings": {
"compressionProperties": null,
"type": "DelimitedTextReadSettings"
},
"storeSettings": {
"enablePartitionDiscovery": false,
"recursive": true,
"type": "AzureBlobFSReadSettings"
},
"type": "DelimitedTextSource"
},
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
},
"mappings": [
{
"sink": {
"name": "sample2_id",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "sample2_id",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column21",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column21",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column22",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column22",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column23",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column23",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column24",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column24",
"physicalType": "String",
"type": "String"
}
},
{
"sink": {
"name": "column25",
"physicalType": "String",
"type": "String"
},
"source": {
"name": "column25",
"physicalType": "String",
"type": "String"
}
}
]
}
},
"userProperties": []
},
{
"dependsOn": [
{
"activity": "CopyToSilverSample1",
"dependencyConditions": [
"Succeeded"
]
},
{
"activity": "CopyToSilverSample2",
"dependencyConditions": [
"Succeeded"
]
}
],
"name": "CreateGoldData",
"policy": {
"retry": 0,
"retryIntervalInSeconds": 30,
"secureInput": false,
"secureOutput": false,
"timeout": "0.12:00:00"
},
"type": "ExecuteDataFlow",
"typeProperties": {
"compute": {
"computeType": "General",
"coreCount": 8
},
"dataFlow": {
"datasetParameters": {
"sinkGold": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": "sample.csv"
},
"sourceSilverSample1": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_1"
}
},
"sourceSilverSample2": {
"container_name": {
"type": "Expression",
"value": "@pipeline().parameters.container_name"
},
"file_name": {
"type": "Expression",
"value": "@pipeline().parameters.sample_file_2"
}
}
},
"parameters": {},
"referenceName": "create_gold_data_sample",
"type": "DataFlowReference"
},
"staging": {},
"traceLevel": "Fine"
},
"userProperties": []
}
]
DataFactory の作成
Data Factory リソースを作成する。
create_datafactory.tf
create_datafactory.tf
resource "azurerm_data_factory" "adf" {
name = local.data_factory.name
resource_group_name = data.azurerm_resource_group.my_rg.name
location = data.azurerm_resource_group.my_rg.location
identity {
type = "SystemAssigned"
}
tags = local.default_tags
}
resource "azurerm_data_factory_linked_service_data_lake_storage_gen2" "gen2_storages" {
for_each = local.storage_account.names
name = "LnkdSrvDataLakeGen2${title(each.key)}"
data_factory_id = azurerm_data_factory.adf.id
url = "https://${azurerm_storage_account.storages[each.key].name}.dfs.core.windows.net/"
use_managed_identity = true
}
resource "azurerm_data_factory_dataset_delimited_text" "storages" {
for_each = local.storage_account.names
name = "DelimitedText${title(each.key)}"
data_factory_id = azurerm_data_factory.adf.id
linked_service_name = azurerm_data_factory_linked_service_data_lake_storage_gen2.gen2_storages[each.key].name
azure_blob_fs_location {
file_system = "@dataset().container_name"
filename = "@dataset().file_name"
}
column_delimiter = ","
row_delimiter = "\n"
encoding = "UTF-8"
first_row_as_header = true
parameters = {
"container_name" = ""
"file_name" = ""
}
}
data "local_file" "sample" {
filename = "pipeline_definitions/sample.json"
}
resource "azurerm_data_factory_pipeline" "sample" {
depends_on = [
azurerm_data_factory_dataset_delimited_text.storages["raw"],
azurerm_data_factory_data_flow.create_gold_data_sample
]
name = "sample"
data_factory_id = azurerm_data_factory.adf.id
parameters = {
"container_name" = "sample"
"sample_file_1" = "sample1.csv"
"sample_file_2" = "sample2.csv"
}
activities_json = data.local_file.sample.content
}
data "local_file" "create_gold_data_sample" {
filename = "pipeline_definitions/create_gold_data_sample.txt"
}
resource "azurerm_data_factory_data_flow" "create_gold_data_sample" {
name = "create_gold_data_sample"
data_factory_id = azurerm_data_factory.adf.id
source {
name = "sourceSilverSample1"
dataset {
name = azurerm_data_factory_dataset_delimited_text.storages["silver"].name
}
}
source {
name = "sourceSilverSample2"
dataset {
name = azurerm_data_factory_dataset_delimited_text.storages["silver"].name
}
}
transformation {
name = "joinSample1Sample2"
}
sink {
name = "sinkGold"
dataset {
name = azurerm_data_factory_dataset_delimited_text.storages["gold"].name
}
}
script = data.local_file.create_gold_data_sample.content
}
ロールのアサイン
ストレージアカウント内のデータアクス用に、ADF のマネージド ID とカレントユーザにロールを割り当てる。割り当てるロールは、Storage Blob Data Contributor。
assign_role.tf
resource "azurerm_role_assignment" "adf_write_storages" {
for_each = local.storage_account.names
scope = azurerm_storage_account.storages[each.key].id
role_definition_name = "Storage Blob Data Contributor"
principal_id = azurerm_data_factory.adf.identity[0].principal_id
}
resource "azurerm_role_assignment" "current_user_write_storages" {
for_each = local.storage_account.names
scope = azurerm_storage_account.storages[each.key].id
role_definition_name = "Storage Blob Data Contributor"
principal_id = data.azurerm_client_config.current.object_id
}
プロビジョニング
以下コマンドでプロビジョニングする。問題なければ今回作成することとしていたパイプラインが作成されているはず。
terraform init
terraform apply -y
動作確認
Debug をクリックして確認する。Raw -> Bronze -> Silver -> Gold の順にデータが処理され、それぞれのストレージアカウント内にもデータが作成されているのが確認できる。
ということで
パイプラインを含めて環境をつくるために、Terraform で各リソースとパイプラインの構築を行いました。今回はハンズオン環境用としたかったので、この方法で行いましたが一般的にパイプラインは Data Factory Studio 上で GUI 操作によって作成する方が分かりやすさや操作面で優れていると思います。Github などとも連携してパイプラインのコードも管理できますし。逆に、Terraform で作成したパイプラインを Git 連携してしまうとめんどくさいことになりそうな気がします (未検証)。
用途は限られてしまいますが、こういった使い方もできるという紹介でした。
以上です。