1
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

[Spark] columnの数が可変なResultSetからDataframeへの変換

Last updated at Posted at 2020-05-20

Javaのsqldriverとかをたたくと、よくResultSetが返ってくるのですが、その際にdataframeに変換したいということがあったので、メモ程度に.

ResultSetで返ってくるtableのcolumnの数があらかじめ分かっていない時です。

    val rs = stmt.getResultSet()

    val schema = (1 to rs.getMetaData().getColumnCount()).map { i =>
      StructField(rs.getMetaData().getColumnName(i), StringType, true)
    }.toList

    val data = Iterator
      .continually { rs }
      .takeWhile(_.next())
      .map { rs =>
        val row = (1 to rs.getMetaData().getColumnCount()).map { i =>
          rs.getString(i)
        }.toList
        Row(row: _*)
      }
      .toList

    spark.createDataFrame(
      spark.sparkContext.parallelize(data),
      StructType(schema)
    )

こうした方がいいなどありましたら、ご教授ください.

1
1
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
1
1

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?