LoginSignup
0

More than 5 years have passed since last update.

Flow の型チェックを CircleCI から実行する時はワーカーの上限値を設定した方がよかった話

Posted at

前提条件

  • CircleCI など仮想環境上での実行であること。
  • flow-bin v0.82.0 以降であること。
  • 型チェックする対象ファイルが多数存在すること。

現象

型チェックの対象となるファイルを大量に置いた状態から Flow の型チェックを実行すると、次のようなエラーメッセージを吐いて停止してしまう。

#!/bin/bash -eo pipefail
npm run flow

> crash-flow-worker-on-virtual-machine@1.0.0 flow /home/circleci/repo
> npx flow

Launching Flow server for /home/circleci/repo
Spawned flow server (pid=136)
Logs will go to /tmp/flow/zShomezScirclecizSrepo.log
Monitor logs will go to /tmp/flow/zShomezScirclecizSrepo.monitor_log
Please wait. Server is starting up: \Please wait. Server is initializing (parsed files 3000): -Please wait. Server is initializing (parsed files 10056): \Please wait. Server is initializing (parsed files 10056): |Please wait. Server is initializing (parsed files 10056): /Please wait. Server is initializing (resolving dependencies): -Please wait. Server is initializing (resolving dependencies): \Please wait. Server is initializing (resolving dependencies): |Please wait. Server is initializing (resolving dependencies): /Please wait. Server is initializing (resolving dependencies): -Please wait. Server is initializing (merged files 0/3821 (0.0%)): -Please wait. Server is initializing (merged files 1387/3821 (36.3%)): \Please wait. Server is initializing (merged files 1764/3821 (46.2%)): |Please wait. Server is initializing (merged files 2036/3821 (53.3%)): /Please wait. Server is initializing (merged files 2283/3821 (59.7%)): -Please wait. Server is initializing (merged files 2352/3821 (61.6%)): \Please wait. Server is initializing (merged files 2475/3821 (64.8%)): |Please wait. Server is initializing (merged files 2585/3821 (67.7%)): /Please wait. Server is initializing (merged files 2693/3821 (70.5%)): -Please wait. Server is initializing (merged files 2772/3821 (72.5%)): \Please wait. Server is initializing (merged files 2881/3821 (75.4%)): |Please wait. Server is initializing (merged files 2991/3821 (78.3%)): /Please wait. Server is initializing (merged files 3067/3821 (80.3%)): -Please wait. Server is initializing (merged files 3153/3821 (82.5%)): \The flow server is not responding (3 retries remaining): -Launching Flow server for /home/circleci/repo
Spawned flow server (pid=688)
Logs will go to /tmp/flow/zShomezScirclecizSrepo.log
Monitor logs will go to /tmp/flow/zShomezScirclecizSrepo.monitor_log
Please wait. Server is initializing (parsed files 0): -Please wait. Server is initializing (parsed files 5000): \Please wait. Server is initializing (parsed files 10056): |Please wait. Server is initializing (parsed files 10056): /Please wait. Server is initializing (resolving dependencies): \Please wait. Server is initializing (resolving dependencies): |Please wait. Server is initializing (resolving dependencies): /Please wait. Server is initializing (resolving dependencies): -Please wait. Server is initializing (resolving dependencies): \Please wait. Server is initializing (merged files 0/3821 (0.0%)): \Please wait. Server is initializing (merged files 1407/3821 (36.8%)): |Please wait. Server is initializing (merged files 1791/3821 (46.9%)): /Please wait. Server is initializing (merged files 2097/3821 (54.9%)): -Please wait. Server is initializing (merged files 2236/3821 (58.5%)): \Please wait. Server is initializing (merged files 2388/3821 (62.5%)): |Please wait. Server is initializing (merged files 2428/3821 (63.5%)): /Please wait. Server is initializing (merged files 2577/3821 (67.4%)): -Please wait. Server is initializing (merged files 2668/3821 (69.8%)): \Please wait. Server is initializing (merged files 2734/3821 (71.6%)): |Please wait. Server is initializing (merged files 2837/3821 (74.2%)): /Please wait. Server is initializing (merged files 2914/3821 (76.3%)): -Please wait. Server is initializing (merged files 3027/3821 (79.2%)): \Please wait. Server is initializing (merged files 3170/3821 (83.0%)): |Please wait. Server is initializing (merged files 3287/3821 (86.0%)): /The flow server is not responding (3 retries remaining): \Launching Flow server for /home/circleci/repo
Spawned flow server (pid=1254)

...

Logs will go to /tmp/flow/zShomezScirclecizSrepo.log
Monitor logs will go to /tmp/flow/zShomezScirclecizSrepo.monitor_log
Please wait. Server is initializing (parsed files 0): |Please wait. Server is initializing (parsed files 5000): /Please wait. Server is initializing (parsed files 10056): -Please wait. Server is initializing (parsed files 10056): \Please wait. Server is initializing (parsed files 10056): |Please wait. Server is initializing (resolving dependencies): /Please wait. Server is initializing (resolving dependencies): -Please wait. Server is initializing (resolving dependencies): \Please wait. Server is initializing (resolving dependencies): |Please wait. Server is initializing (waiting for Watchman - giving up in 14 seconds): -Please wait. Server is initializing (merged files 0/3821 (0.0%)): /Please wait. Server is initializing (merged files 1447/3821 (37.9%)): -Please wait. Server is initializing (merged files 1817/3821 (47.6%)): \Please wait. Server is initializing (merged files 2074/3821 (54.3%)): |Please wait. Server is initializing (merged files 2269/3821 (59.4%)): /Please wait. Server is initializing (merged files 2385/3821 (62.4%)): -Please wait. Server is initializing (merged files 2488/3821 (65.1%)): \Please wait. Server is initializing (merged files 2615/3821 (68.4%)): |Please wait. Server is initializing (merged files 2692/3821 (70.5%)): /Please wait. Server is initializing (merged files 2780/3821 (72.8%)): -Please wait. Server is initializing (merged files 2864/3821 (75.0%)): \Please wait. Server is initializing (merged files 2941/3821 (77.0%)): |Please wait. Server is initializing (merged files 3043/3821 (79.6%)): /Lost connection to the flow server (0 retries remaining): \Out of retries, exiting!
npm ERR! code ELIFECYCLE
npm ERR! errno 7
npm ERR! crash-flow-worker-on-virtual-machine@1.0.0 flow: `npx flow`
npm ERR! Exit status 7
npm ERR! 
npm ERR! Failed at the crash-flow-worker-on-virtual-machine@1.0.0 flow script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2019-02-02T09_12_20_617Z-debug.log
Exited with code 7

一方で手元の環境では問題なく動作する。

ここで Flow のコマンドを flow check に修正し、 CircleCI 上で再度実行すると次のようなエラーメッセージを吐くようになった。

#!/bin/bash -eo pipefail
npm run flow

> crash-flow-worker-on-virtual-machine@1.0.0 flow /home/circleci/repo
> npx flow check


Worker interrupted with signal: sigkill
(709) merge_strict_job THROWS: [1] /home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactCurrentFiber.js: Utils_js.Key_not_found("LeaderHeap", "/home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactFiber.js")
Raised at file "hashtbl.ml", line 194, characters 13-28
Called from file "hack/heap/sharedMem.ml", line 1136, characters 22-42


(709) merge_strict_job THROWS: [1] /home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactDebugFiberPerf.js: Utils_js.Key_not_found("LeaderHeap", "/home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactFiber.js")
Raised at file "hashtbl.ml", line 194, characters 13-28
Called from file "hack/heap/sharedMem.ml", line 1136, characters 22-42

...

Worker slave 690 exception: Unix.Unix_error(Unix.EPIPE, "write", "")
Worker slave 690 Potential backtrace:
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 145, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 158, characters 4-63
Called from file "hack/utils/measure.ml", line 238, characters 12-16
Called from file "hack/procs/worker.ml", line 88, characters 14-143
Called from file "hack/procs/worker.ml", line 128, characters 4-37

Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 145, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 158, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
(708) merge_strict_job THROWS: [1] /home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactProfilerTimer.js: Utils_js.Key_not_found("LeaderHeap", "/home/circleci/repo/src/copies/3/packages/react-reconciler/src/ReactFiberHostConfig.js")
Raised at file "hashtbl.ml", line 194, characters 13-28
Called from file "hack/heap/sharedMem.ml", line 1136, characters 22-42


Worker slave 708 exception: Unix.Unix_error(Unix.EPIPE, "write", "")
Worker slave 708 Potential backtrace:
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 145, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 158, characters 4-63
Called from file "hack/utils/measure.ml", line 238, characters 12-16
Called from file "hack/procs/worker.ml", line 88, characters 14-143
Called from file "hack/procs/worker.ml", line 128, characters 4-37

npm ERR! code ELIFECYCLE
npm ERR! errno 110
npm ERR! crash-flow-worker-on-virtual-machine@1.0.0 flow: `npx flow check`
npm ERR! Exit status 110
npm ERR! 
npm ERR! Failed at the crash-flow-worker-on-virtual-machine@1.0.0 flow script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2019-02-02T09_15_56_449Z-debug.log
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 145, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 158, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Exited with code 110

原因

この issue のコメント がわかりやすい。

Flow のオプションである server.max_workers についてはドキュメントに次のような記載がある。

server.max_workers (integer)
The maximum number of workers the Flow server can start. By default, the server will use all available cores.
https://flow.org/en/docs/config/options/#toc-server-max-workers-integer

Flow は型チェックする際にサーバーとワーカーを起動するが、デフォルトで起動可能なワーカーの上限数は CPU のコア数としている。しかし仮想環境化では実際に利用可能な CPU のコア数よりも多いワーカーを起動可能となってしまうため、メモリを食い潰してプロセスが強制終了されてしまうケースがあるようだ。

対策

issue 付いたコメントのように server.max_workers をパフォーマンス上許容できる値に指定すると良い。私の環境では 1 だと時間が掛かり過ぎるので 4 にした。

[options]
server.max_workers=4

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
0