TerraformでBigQuery Data Transfer Service for Amazon S3を設定する

data "google_project" "project" {
}

resource "google_project_iam_member" "permissions" {
  role   = "roles/iam.serviceAccountShortTermTokenMinter"
  member = "serviceAccount:service-${data.google_project.project.number}@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.com"
}

resource "google_bigquery_data_transfer_config" "s3-transfer" {
  depends_on             = [google_project_iam_member.permissions]

  display_name           = "s3-transfer"
  location               = "asia-northeast1"
  data_source_id         = "amazon_s3"
  schedule               = "every day 21:00" # UTC
  destination_dataset_id = google_bigquery_dataset.my_dataset.dataset_id
  params                 = {
    destination_table_name_template = "my_table"
    data_path                       = "s3://my_bucket/my_prefix/*"
    access_key_id                   = var.aws_access_key
    secret_access_key               = var.aws_secret_key
    file_format                     = "PARQUET"
  }
}

Terraformのドキュメントにはs3転送例は載っていないためdata_source_idやparamsの項目は個別に調べる必要がある。BigQuery Data Transfer Service がサポートするデータソースはドキュメント記載のコマンド出力から得られる。