29
23

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

Google BigQuery テーブルスキーマ変更したいです。

Posted at

アプリからイベントデータをガンガン送信するぜ!
BigQuery上にBigData載せてデータ解析するぜ!
テーブル作ったぜ!

イベントデータのプロパティに情報追加したいぜ!
BigQuery loadできないぜ!?

BigQuery error in load operation: Illegal Schema update. Cannot add fields (field: hogehoge)

そんな時は下記コマンドでちゃんとスキーマアップデートできるんです。

bq update db_name.table_name table.json

でもtable.jsonって何書くのよって話ですが、既存のカラムも書いた上で追加したいもの追加したり変更したりしないといけないのです。
なので、カラムたくさんあったりするときは大変です。
というわけで下記で今のテーブルのスキーマとってきましょう。

bq --format=prettyjson show db_name.table_name > table.json

それでtable.jsonの中身を見てみると、

{
  "creationTime": "000000000",
  "etag": "\"MyUncleIsAGoodTennisPlayer\"",
  "id": "MyUncle:db_name.table_name",
  "kind": "bigquery#table",
  "lastModifiedTime": "999999999999",
  "numBytes": "12",
  "numRows": "123123123",
  "schema": {
    "fields": [
      {
        "mode": "NULLABLE",
        "name": "event",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "mode": "NULLABLE",
            "name": "time",
            "type": "TIMESTAMP"
          },
          {
            "mode": "NULLABLE",
            "name": "distinct_id",
            "type": "STRING"
          },
          ...
          {
            "mode": "NULLABLE",
            "name": "uncle",
            "type": "BOOLEAN"
          }
        ],
        "mode": "NULLABLE",
        "name": "properties",
        "type": "RECORD"
      }
    ]
  },
  "selfLink": "https://www.googleapis.com/bigquery/v2/projects/myuncle/datasets/db_name/tables/table_name",
  "tableReference": {
    "datasetId": "db_name",
    "projectId": "myuncleisagoodtennisplayer",
    "tableId": "table_name"
  },
  "type": "TABLE"
}

こんなjsonがかえってくるので、こいつを流用します。

といってもそのまま使うと怒られるので、この部分だけ残します。
コンマの後ろの半角スペースがなかったらそれもまた怒られるので、消さないようにしましょう。
エディタの設定で保存時に末尾の空白を削除する設定はないでしょうか。

    [
      {
        "mode": "NULLABLE",
        "name": "event",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "mode": "NULLABLE",
            "name": "time",
            "type": "TIMESTAMP"
          },
          {
            "mode": "NULLABLE",
            "name": "distinct_id",
            "type": "STRING"
          },
          ...
          {
            "mode": "NULLABLE",
            "name": "uncle",
            "type": "BOOLEAN"
          }, 
          {
            "mode": "NULLABLE",
            "name": "あたらしいカラム!!!!!!!!",
            "type": "STRING"
          }
        ],
        "mode": "NULLABLE",
        "name": "properties",
        "type": "RECORD"
      }
    ]

そしてupdate。

bq update db_name.table_name table.json                                             
Table 'myuncleisagoodtennisplayer:db_name.table_name' successfully updated.
29
23
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
29
23

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?