Embulk

Embulkでタスクを処理するスレッドの数を変更する方法

More than 3 years have passed since last update.

タスクを処理するスレッドの数を変更するには、システムプロパティで

-Dembulk.max_threads=1

のように指定すれば良い模様。

>set JAVA_TOOL_OPTIONS=-Dembulk.max_threads=1

>embulk run -I lib config.yml
Picked up JAVA_TOOL_OPTIONS: -Dembulk.max_threads=1
2015-03-11 14:14:30.523 +0900: Embulk v0.5.1
2015-03-11 14:14:33.959 +0900 [INFO] (transaction): {done:  0 / 4, running: 0}
2015-03-11 14:14:33.990 +0900 [INFO] (task-0000): Task #0 is running...
John,10
John,30
2015-03-11 14:14:37.178 +0900 [INFO] (transaction): {done:  1 / 4, running: 0}
2015-03-11 14:14:37.178 +0900 [INFO] (task-0001): Task #1 is running...
Jane,5
Jane,100
2015-03-11 14:14:40.180 +0900 [INFO] (transaction): {done:  2 / 4, running: 0}
2015-03-11 14:14:40.182 +0900 [INFO] (task-0002): Task #2 is running...
Tom,200
Tom,80
2015-03-11 14:14:43.184 +0900 [INFO] (transaction): {done:  3 / 4, running: 0}
2015-03-11 14:14:43.186 +0900 [INFO] (task-0003): Task #3 is running...
Mary,20
Mary,50
2015-03-11 14:14:46.188 +0900 [INFO] (transaction): {done:  4 / 4, running: 0}
2015-03-11 14:14:46.268 +0900 [INFO] (main): Committed.
2015-03-11 14:14:46.268 +0900 [INFO] (main): Next config diff: {"in":{},"out":{}}

>set JAVA_TOOL_OPTIONS=-Dembulk.max_threads=4

>embulk run -I lib config.yml
Picked up JAVA_TOOL_OPTIONS: -Dembulk.max_threads=4
2015-03-11 14:15:10.892 +0900: Embulk v0.5.1
2015-03-11 14:15:14.044 +0900 [INFO] (transaction): {done:  0 / 4, running: 0}
2015-03-11 14:15:14.085 +0900 [INFO] (task-0001): Task #1 is running...
2015-03-11 14:15:14.085 +0900 [INFO] (task-0002): Task #2 is running...
2015-03-11 14:15:14.085 +0900 [INFO] (task-0003): Task #3 is running...
2015-03-11 14:15:14.085 +0900 [INFO] (task-0000): Task #0 is running...
Tom,200
John,10
John,30
Jane,5
Jane,100
Mary,20
Mary,50
Tom,80
2015-03-11 14:15:17.348 +0900 [INFO] (transaction): {done:  4 / 4, running: 0}
2015-03-11 14:15:17.349 +0900 [INFO] (transaction): {done:  4 / 4, running: 0}
2015-03-11 14:15:17.350 +0900 [INFO] (transaction): {done:  4 / 4, running: 0}
2015-03-11 14:15:17.350 +0900 [INFO] (transaction): {done:  4 / 4, running: 0}
2015-03-11 14:15:17.377 +0900 [INFO] (main): Committed.
2015-03-11 14:15:17.377 +0900 [INFO] (main): Next config diff: {"in":{},"out":{}
}

参考資料