build した beam で iex/mix を実行、coredump を解析 #Elixir

clone

https://github.com/erlang/otp.git は ~/Repos/otp にクローン済みでスタート

git clone ~/Repos/otp otp-26.1.2 -b OTP-26.1.2
cd otp-26.1.2

configure

export ERL_TOP=`pwd`
# ./configure -h で オプションは確認できる
./configure --enable-kernel-poll --enable-dirty-schedulers --enable-jit --enable-debug-mode

make

Compile Server
CPU 全部 -j

を使って make

ERLC_USE_SERVER=true make -j

beam.debug.smp の make

make -j -C erts/emulator debug

生成物の確認

ls bin/
cerl    dialyzer  erl_call  escript             no_dot_erlang.script  start.script      start_clean.script  start_sasl.script  x86_64-pc-linux-gnu
ct_run  erl       erlc      no_dot_erlang.boot  start.boot            start_clean.boot  start_sasl.boot     typer

ls bin/x86_64-pc-linux-gnu/
beam.debug.jit  beam.debug.smp  beam.jit  beam.smp  ct_run  dialyzer  dyn_erl  epmd  erl_call  erl_child_setup  erl_child_setup.debug  erlc  erlexec  escript  heart  inet_gethost  jit-reader.so  libbeam.a  run_erl  to_erl  typer

iex コマンドを確かめる

ELIXIR_CLI_DRY_RUN=1 iex
erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -user elixir -extra --no-halt +iex

build した beam.smp で iex を実行する

./bin/erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -user elixir -extra --no-halt +iex

Erlang/OTP 26 [erts-14.1.1] [source] [64-bit] [smp:32:32] [ds:32:32:10] [async-threads:1] [jit:ns]

Interactive Elixir (1.15.7) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

iex -S mix の実行でエラー & 回避

エラー

/home/pojiro/Sandbox/otp-26.1.2/bin/erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -user elixir -extra --no-halt +iex -S mix
** (SyntaxError) /home/pojiro/.asdf/shims/mix:6:19: syntax error before: '.'
    |
  6 | exec /home/pojiro/.asdf/bin/asdf exec "mix" "$@" # asdf_allow: ' asdf '
    |                   ^
    (elixir 1.15.7) lib/code.ex:1435: Code.require_file/2

回避

mix をフルパスで与えると回避できるが、原因不明

ref. https://github.com/asdf-vm/asdf/issues/926#issuecomment-1890830438

/home/pojiro/Sandbox/otp-26.1.2/bin/erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -user elixir -extra --no-halt +iex -S /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/mix
Erlang/OTP 26 [erts-14.1.1] [source] [64-bit] [smp:32:32] [ds:32:32:10] [async-threads:1] [jit:ns]

Interactive Elixir (1.15.7) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

beam の coredump をとって、 gdb で解析できるようにする

前提： mix test が core を吐く、そのプロジェクトのディレクトリには cd 済み

コマンドを調べる

ELIXIR_CLI_DRY_RUN=1 mix test test/test_defer_load_nif_test.exs:10
erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_cli -elixir ansi_enabled true -extra /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/mix test test/test_defer_load_nif_test.exs:10

build した beam.smp で実行する

/home/pojiro/Sandbox/otp-26.1.2/bin/erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -extra /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/mix test test/test_defer_load_nif_test.exs:10
...
Segmentation fault (コアダンプ)

build した beam.debug.smp で実行する

erl -> cerl -debug 点に注意

/home/pojiro/Sandbox/otp-26.1.2/bin/cerl -debug -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -extra /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/mix test test/test_defer_load_nif_test.exs:10
...
# コア吐かなくなった。。ぬー

コアを探す

cat /proc/sys/kernel/core_pattern 
|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h

systemd-coredump にパイプされているので、/var/lib/systemd/coredump/ で探す。

ls -la /var/lib/systemd/coredump/
合計 83244
drwxr-xr-x   2 root root     4096  1月 16 20:34  .
drwxr-xr-x  12 root root     4096  1月  3 21:16  ..
-rw-r-----+  1 root root 17119855  1月 16 20:22 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.5858.1705404135000000.zst'
-rw-r-----+  1 root root 16890586  1月 16 20:24 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.6757.1705404273000000.zst'
-rw-r-----+  1 root root 16880366  1月 16 20:25 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.7049.1705404310000000.zst'
-rw-r-----+  1 root root 16995351  1月 16 20:25 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.7377.1705404313000000.zst'
-rw-r-----+  1 root root 17315907  1月 16 20:34 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.8778.1705404843000000.zst'

zstを解凍

cp /var/lib/systemd/coredump/'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.8778.1705404843000000.zst' .
zstd -d 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.8778.1705404843000000.zst'

cerl -rcore で gdb 起動

beam.smp で実行したコアの解析は -debug オプション無しでやる
beam.debug.smp で実行したコアの解析は -debug オプション有りでやる

記事を書くきっかけの issue は beam.debug.smp だとコアダンプしなかったので、 beam.smp で実行したコアの解析をする

/home/pojiro/Sandbox/otp-26.1.2/bin/cerl -rcore core.beam\\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.8778.1705404843000000

起動すると、

Core was generated by `/home/pojiro/Sandbox/otp-26.1.2/bin/x86_64-pc-linux-gnu/beam.smp -- -root /home'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Section `.reg-xstate/5449' in core file too small.
#0  0x00007fa935683cc8 in ?? ()
[Current thread is 1 (Thread 0x7fa93251e640 (LWP 5449))]
%---------------------------------------------------------------------------
% Use etp-help for a command overview and general help.
%
% To use the Erlang support module, the environment variable ROOTDIR
% must be set to the toplevel installation directory of Erlang/OTP,
% so the etp-commands file becomes:
%     $ROOTDIR/erts/etc/unix/etp-commands
% Also, erl and erlc must be in the path.
%---------------------------------------------------------------------------
etp-set-max-depth 20
etp-set-max-string-length 100
--------------- System Information ---------------
OTP release: 26
ERTS version: 14.1.1
Arch: x86_64-pc-linux-gnu
Endianness: Little
Word size: 64-bit
BeamAsm support: yes
SMP support: yes
Thread support: yes
Kernel poll: Supported and used
Debug compiled: no
Lock checking: no
Lock counting: no
Node name: nonode@nohost
Number of schedulers: 32
Number of async-threads: 1
--------------------------------------------------
(gdb)

参考

読めてないけど

How to analyse a BEAM core dump?