Why not login to Qiita and try out its useful features?

We'll deliver articles that match you.

You can read useful information later.

4
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 1 year has passed since last update.

ZOZOAdvent Calendar 2023

Day 8

BigQuery → TensorFlow Dataset

Last updated at Posted at 2023-12-07

本記事は ZOZO Advent Calendar 2023 シリーズ 7 の 8 日目の記事です。

概要

BigQuery のテーブルを TensorFlow Dataset に変換する方法を2つご紹介します。

1. BigQuery リーダーを使用する方法

チュートリアル記載の方法です。

import tensorflow as tf
from tensorflow_io.bigquery import BigQueryClient
from google.cloud import bigquery

PROJECT_ID = "project_id"
DATASET_ID = "dataset_id"
CSV_SCHEMA = [
      bigquery.SchemaField("user_id", "INTEGER"),
      bigquery.SchemaField("item_id", "INTEGER"),
      bigquery.SchemaField("score", "INTEGER"),
]

def read_bigquery(table_name):
    tensorflow_io_bigquery_client = BigQueryClient()
    read_session = tensorflow_io_bigquery_client.read_session(
        "projects/" + PROJECT_ID,
        PROJECT_ID,
        table_name,
        DATASET_ID,
        list(field.name for field in CSV_SCHEMA),
        requested_streams=2,
    )

    dataset = read_session.parallel_read_rows()
    return dataset

train = (
    read_bigquery("table_name")
    .batch(256)
)

2. Cloud Storage を介する方法

以下の方法は Cloud Storage に中間データを保存したい場合に使えます。

1. BigQuery → Cloud Storage

EXPORT DATA
  OPTIONS (
    uri = "gs://bucket_name/dataset/*.csv",
    format = "CSV",
    overwrite = true,
    header = true
  )
AS 
SELECT
  user_id,
  item_id,
  score,
FROM `project_id.dataset_id.table_name`

2. Cloud Storage → TensorFlow Dataset

import tensorflow as tf
train_data = tf.data.experimental.make_csv_dataset(
    file_pattern="gs://bucket_name/dataset/*.csv",
    batch_size=256,
)
4
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up

Qiita Advent Calendar is held!

Qiita Advent Calendar is an article posting event where you post articles by filling a calendar 🎅

Some calendars come with gifts and some gifts are drawn from all calendars 👀

Please tie the article to your calendar and let's enjoy Christmas together!

4
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?