Go to Qiita Advent Calendar 2024 Top
LoginSignup
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@manabian

Databricks（Spark）にてPysparkによりコア数（スロット数）を確認する方法

Last updated at Posted at 2021-08-11

概要

Databricks(Spark)にてPysparkによりコア数（スロット数）を確認する方法を共有します。

実行環境

databricks runtime: 8.3.x-cpu-ml-scala2.12
Python version: 3.8.8
pyspark version: 3.1.2.dev0

手順

下記のコードを実行。

spark.sparkContext.defaultParallelism

ワーカーノードにて4CoresのVMが1台の場合には、4と表示

実行結果
image.png

ワーカーノードのVMの設定
image.png

ワーカーノードにて4CoresのVMが3台の場合には、12と表示。

実行結果
image.png

ワーカーノードのVMの設定
image.png

1
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?