Help us understand the problem. What is going on with this article?

BigQueryで、CROSS JOINを使わずに配列から重複を省く

More than 1 year has passed since last update.

はじめに

BigQueryで、配列から重複を省く方法です。

シンプルな場合

重複アイテムを持つ配列内の重複を削除します。

#standardSQL
WITH T_SAMPLE AS(
  SELECT 'id1' id, ['item1', 'item2', 'item2'] items
  UNION ALL SELECT 'id2', ['item1', 'item2']
  UNION ALL SELECT 'id3', ['item10', 'item11', 'item10']
)

SELECT 
  id,
  (SELECT ARRAY_AGG(DISTINCT items_list) FROM UNNEST(T_SAMPLE.items) items_list) items
FROM 
  T_SAMPLE
クエリ実行前のテーブル 実行後のテーブル
T_SAMPLE
スクリーンショット 2018-04-20 6.31.54.png スクリーンショット 2018-04-20 6.28.27.png

集約しつつ重複を省く場合

サブクエリでいったん配列を結合してから、同様に集約します。(しか無い?)

#standardSQL
WITH T_SAMPLE AS(
  SELECT 'id1' id, ['item1', 'item2', 'item2'] items
  UNION ALL SELECT 'id1', ['item1', 'item2']
  UNION ALL SELECT 'id3', ['item10', 'item11', 'item10']
)

SELECT 
  id,
  (SELECT ARRAY_AGG(DISTINCT items_list) FROM UNNEST(items) items_list) items
FROM(
  SELECT 
    id,
    ARRAY_CONCAT_AGG(T_SAMPLE.items) items
  FROM 
    T_SAMPLE
  GROUP BY
    id
)
クエリ実行前のテーブル 実行後のテーブル
T_SAMPLE
スクリーンショット 2018-04-20 6.53.59.png スクリーンショット 2018-04-20 6.53.14.png

おわりに

最近は、AWS Athenaも使っていきたいです。

Bacchus
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした