目的・概要
サーバのパフォーマンステスト等で意図的にメモリ消費した状態にしたいことがあります。
これをある程度簡単に実現するための方法をメモします。
TL;DR
Python2で、range関数を利用してメモリを大量に消費するスクリプトを作ります。
実装する
検証した環境(AWS)
EC2 instance size: m3.medium
OS: Amazon Linux AMI 2017.09
Python: 2.7.12(※)
※Python3のrange関数の実装はイテレータを返すように変更されているので、本稿の方法でメモリを大量に消費させることはできないと思います。
メモリ消費スクリプト(greedy_memory_consumer.py)
greedy_memory_consumer.py
#!/usr/bin/python
import time
item_num_readable = '108,000,000'
huge_items = range(int(item_num_readable.replace(',','')))
while True:
print ("sleep.")
time.sleep(60)
動作確認
psコマンドによる確認
およそ3.4GB程度のメモリを消費していることがわかります。
$ ps -elfy | grep pytho[n]
S ec2-user 2790 2738 1 80 0 3442688 891043 poll_s 15:36 pts/1 00:00:03 python greedy_memory_consumer.py
topコマンドによる確認
メモリ4GBのサーバで実行した結果です。
%MEMが89.3となっていることが確認できます。
# topコマンドで、メモリ利用率順に並べる
# top -a
top - 15:28:10 up 9 min, 3 users, load average: 0.00, 0.01, 0.00
Tasks: 84 total, 1 running, 82 sleeping, 1 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3855812k total, 3541800k used, 314012k free, 5012k buffers
Swap: 0k total, 0k used, 0k free, 29152k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2790 ec2-user 20 0 3480m 3.3g 3664 S 0.0 89.3 0:03.71 python
2297 root 20 0 298m 13m 7676 S 0.0 0.4 0:00.29 amazon-ssm-agen
2789 ec2-user 20 0 151m 8548 5512 T 0.0 0.2 0:00.05 vim
2738 ec2-user 20 0 112m 2848 2412 S 0.0 0.1 0:00.02 bash
2756 ec2-user 20 0 112m 2752 2336 S 0.0 0.1 0:00.01 bash
**snipped**
killedが出たら
スクリプト実行後、下記のように「Killed」と出力される場合があります。
これは、メモリを消費しすぎてOOM-Killerに殺されているからです。
$ python greedy_memory_consumer.py
Killed
dmesgで確認してみましょう。
# dmesg
[ 333.027337] python invoked oom-killer: gfp_mask=0x24280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=0, order=0, oom_score_adj=0
[ 333.034628] python cpuset=/ mems_allowed=0
[ 333.038319] CPU: 0 PID: 2785 Comm: python Tainted: G E 4.9.62-21.56.amzn1.x86_64 #1
[ 333.040557] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[ 333.040557] ffffc90002a47b50 ffffffff8130225c ffffc90002a47ce8 ffff8800e98bbb80
[ 333.040557] ffffc90002a47be0 ffffffff811fbf7b 0000000000000000 0000000000000000
[ 333.040557] ffffc90002a47c08 ffffc90002a47b88 ffffffff8112a03a ffffc90002a47bf0
[ 333.040557] Call Trace:
[ 333.040557] [<ffffffff8130225c>] dump_stack+0x63/0x87
[ 333.040557] [<ffffffff811fbf7b>] dump_header+0x82/0x212
[ 333.040557] [<ffffffff8112a03a>] ? __delayacct_freepages_end+0x2a/0x30
[ 333.040557] [<ffffffff8119576a>] ? do_try_to_free_pages+0x2da/0x340
[ 333.040557] [<ffffffff8118172c>] oom_kill_process+0x21c/0x3f0
[ 333.040557] [<ffffffff81181bc8>] out_of_memory+0x108/0x4b0
[ 333.040557] [<ffffffff811866f8>] __alloc_pages_slowpath+0x998/0xb80
[ 333.040557] [<ffffffff81186ac3>] __alloc_pages_nodemask+0x1e3/0x250
[ 333.040557] [<ffffffff811d6f95>] alloc_pages_vma+0xa5/0x220
[ 333.040557] [<ffffffff811b3514>] handle_mm_fault+0x1034/0x13a0
[ 333.040557] [<ffffffff81061015>] __do_page_fault+0x225/0x4a0
[ 333.040557] [<ffffffff810612b2>] do_page_fault+0x22/0x30
[ 333.040557] [<ffffffff815436b8>] page_fault+0x28/0x30
[ 333.118278] Mem-Info:
[ 333.120125] active_anon:938017 inactive_anon:12 isolated_anon:0
active_file:19 inactive_file:37 isolated_file:0
unevictable:0 dirty:9 writeback:0 unstable:0
slab_reclaimable:2059 slab_unreclaimable:2379
mapped:39 shmem:15 pagetables:2536 bounce:0
free:14966 free_pcp:37 free_cma:0
[ 333.164906] Node 0 active_anon:3752068kB inactive_anon:48kB active_file:76kB inactive_file:148kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:156kB dirty:36kB writeback:0kB shmem:60kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
[ 333.182929] Node 0 DMA free:15112kB min:184kB low:228kB high:272kB active_anon:788kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 333.202151] lowmem_reserve[]: 0 3732 3732 3732
[ 333.207286] Node 0 DMA32 free:44752kB min:44868kB low:56084kB high:67300kB active_anon:3751264kB inactive_anon:48kB active_file:76kB inactive_file:148kB unevictable:0kB writepending:36kB present:3915776kB managed:3839908kB mlocked:0kB slab_reclaimable:8236kB slab_unreclaimable:9516kB kernel_stack:1776kB pagetables:10140kB bounce:0kB free_pcp:148kB local_pcp:148kB free_cma:0kB
[ 333.228662] lowmem_reserve[]: 0 0 0 0
[ 333.233662] Node 0 DMA: 2*4kB (UM) 2*8kB (UM) 1*16kB (U) 1*32kB (M) 3*64kB (UM) 2*128kB (UM) 1*256kB (U) 0*512kB 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15112kB
[ 333.252627] Node 0 DMA32: 84*4kB (UME) 52*8kB (UE) 54*16kB (UME) 44*32kB (UE) 24*64kB (UE) 30*128kB (UME) 6*256kB (UE) 6*512kB (UE) 3*1024kB (U) 4*2048kB (UME) 5*4096kB (U) = 44752kB
[ 333.273288] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 333.279764] 70 total pagecache pages
[ 333.282568] 0 pages in swap cache
[ 333.285191] Swap cache stats: add 0, delete 0, find 0/0
[ 333.288835] Free swap = 0kB
[ 333.291240] Total swap = 0kB
[ 333.293614] 982941 pages RAM
[ 333.295986] 0 pages HighMem/MovableOnly
[ 333.298865] 18988 pages reserved
[ 333.301325] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 333.307110] [ 1576] 0 1576 2732 108 11 3 0 -1000 udevd
[ 333.312629] [ 1919] 0 1919 27274 59 21 3 0 0 lvmetad
[ 333.318121] [ 1928] 0 1928 6788 48 16 3 0 0 lvmpolld
[ 333.324496] [ 2128] 0 2128 2342 121 10 3 0 0 dhclient
[ 333.330541] [ 2255] 0 2255 2342 127 9 3 0 0 dhclient
[ 333.336664] [ 2297] 0 2297 76516 1442 32 5 0 0 amazon-ssm-agen
[ 333.343142] [ 2305] 0 2305 13240 96 25 3 0 -1000 auditd
[ 333.349157] [ 2337] 0 2337 61847 83 25 3 0 0 rsyslogd
[ 333.355394] [ 2359] 0 2359 1619 26 9 3 0 0 rngd
[ 333.360335] [ 2377] 32 2377 8830 105 23 3 0 0 rpcbind
[ 333.365483] [ 2398] 29 2398 9972 207 24 3 0 0 rpc.statd
[ 333.370496] [ 2429] 81 2429 5450 57 14 3 0 0 dbus-daemon
[ 333.376066] [ 2464] 0 2464 1088 41 8 3 0 0 acpid
[ 333.380732] [ 2564] 0 2564 20123 207 41 3 0 -1000 sshd
[ 333.385566] [ 2574] 38 2574 7443 149 19 3 0 0 ntpd
[ 333.390500] [ 2594] 0 2594 22384 430 46 3 0 0 sendmail
[ 333.396933] [ 2603] 51 2603 20246 369 41 3 0 0 sendmail
[ 333.403230] [ 2615] 0 2615 30402 157 16 3 0 0 crond
[ 333.409497] [ 2629] 0 2629 4786 42 14 3 0 0 atd
[ 333.415497] [ 2638] 0 2638 29468 272 60 3 0 0 sshd
[ 333.421399] [ 2656] 0 2656 1616 37 8 3 0 0 agetty
[ 333.427109] [ 2657] 0 2657 1079 31 8 3 0 0 mingetty
[ 333.432890] [ 2659] 0 2659 1079 31 7 3 0 0 mingetty
[ 333.439658] [ 2661] 0 2661 1079 29 8 3 0 0 mingetty
[ 333.445990] [ 2664] 0 2664 1079 30 8 3 0 0 mingetty
[ 333.451017] [ 2667] 0 2667 1079 30 8 3 0 0 mingetty
[ 333.455913] [ 2669] 0 2669 1079 30 8 3 0 0 mingetty
[ 333.463895] [ 2671] 0 2671 2731 101 10 3 0 -1000 udevd
[ 333.468825] [ 2672] 0 2672 2731 101 10 3 0 -1000 udevd
[ 333.473523] [ 2688] 500 2688 29468 265 59 3 0 0 sshd
[ 333.478167] [ 2689] 500 2689 28840 100 15 3 0 0 bash
[ 333.483030] [ 2736] 500 2736 31867 70 19 3 0 0 screen
[ 333.488037] [ 2737] 500 2737 32002 236 18 3 0 0 screen
[ 333.492811] [ 2738] 500 2738 28840 116 14 3 0 0 bash
[ 333.497673] [ 2756] 500 2756 28840 110 14 3 0 0 bash
[ 333.503036] [ 2785] 500 2785 974688 932682 1839 6 0 0 python
[ 333.507934] [ 2786] 500 2786 3796 73 14 3 0 0 top
[ 333.512691] Out of memory: Kill process 2785 (python) score 969 or sacrifice child
[ 333.519160] Killed process 2785 (python) total-vm:3898752kB, anon-rss:3730620kB, file-rss:108kB, shmem-rss:0kB
[ 333.751653] oom_reaper: reaped process 2785 (python), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB