目的・概要

サーバのパフォーマンステスト等で意図的にメモリ消費した状態にしたいことがあります。
これをある程度簡単に実現するための方法をメモします。

TL;DR

Python2で、range関数を利用してメモリを大量に消費するスクリプトを作ります。

実装する

検証した環境(AWS)

EC2 instance size: m3.medium
OS: Amazon Linux AMI 2017.09
Python: 2.7.12(※)

※Python3のrange関数の実装はイテレータを返すように変更されているので、本稿の方法でメモリを大量に消費させることはできないと思います。

メモリ消費スクリプト(greedy_memory_consumer.py)

greedy_memory_consumer.py

#!/usr/bin/python

import time

item_num_readable = '108,000,000'
huge_items = range(int(item_num_readable.replace(',','')))

while True:
    print ("sleep.")
    time.sleep(60)

動作確認

psコマンドによる確認

およそ3.4GB程度のメモリを消費していることがわかります。

$ ps -elfy | grep pytho[n]
S ec2-user  2790  2738  1  80   0 3442688 891043 poll_s 15:36 pts/1 00:00:03 python greedy_memory_consumer.py

topコマンドによる確認

メモリ4GBのサーバで実行した結果です。
%MEMが89.3となっていることが確認できます。

# topコマンドで、メモリ利用率順に並べる
# top -a
top - 15:28:10 up 9 min,  3 users,  load average: 0.00, 0.01, 0.00
Tasks:  84 total,   1 running,  82 sleeping,   1 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3855812k total,  3541800k used,   314012k free,     5012k buffers
Swap:        0k total,        0k used,        0k free,    29152k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2790 ec2-user  20   0 3480m 3.3g 3664 S  0.0 89.3   0:03.71 python
 2297 root      20   0  298m  13m 7676 S  0.0  0.4   0:00.29 amazon-ssm-agen
 2789 ec2-user  20   0  151m 8548 5512 T  0.0  0.2   0:00.05 vim
 2738 ec2-user  20   0  112m 2848 2412 S  0.0  0.1   0:00.02 bash
 2756 ec2-user  20   0  112m 2752 2336 S  0.0  0.1   0:00.01 bash
**snipped**

killedが出たら

スクリプト実行後、下記のように「Killed」と出力される場合があります。
これは、メモリを消費しすぎてOOM-Killerに殺されているからです。

$ python greedy_memory_consumer.py
Killed

dmesgで確認してみましょう。

# dmesg
[  333.027337] python invoked oom-killer: gfp_mask=0x24280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=0, order=0, oom_score_adj=0
[  333.034628] python cpuset=/ mems_allowed=0
[  333.038319] CPU: 0 PID: 2785 Comm: python Tainted: G            E   4.9.62-21.56.amzn1.x86_64 #1
[  333.040557] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[  333.040557]  ffffc90002a47b50 ffffffff8130225c ffffc90002a47ce8 ffff8800e98bbb80
[  333.040557]  ffffc90002a47be0 ffffffff811fbf7b 0000000000000000 0000000000000000
[  333.040557]  ffffc90002a47c08 ffffc90002a47b88 ffffffff8112a03a ffffc90002a47bf0
[  333.040557] Call Trace:
[  333.040557]  [<ffffffff8130225c>] dump_stack+0x63/0x87
[  333.040557]  [<ffffffff811fbf7b>] dump_header+0x82/0x212
[  333.040557]  [<ffffffff8112a03a>] ? __delayacct_freepages_end+0x2a/0x30
[  333.040557]  [<ffffffff8119576a>] ? do_try_to_free_pages+0x2da/0x340
[  333.040557]  [<ffffffff8118172c>] oom_kill_process+0x21c/0x3f0
[  333.040557]  [<ffffffff81181bc8>] out_of_memory+0x108/0x4b0
[  333.040557]  [<ffffffff811866f8>] __alloc_pages_slowpath+0x998/0xb80
[  333.040557]  [<ffffffff81186ac3>] __alloc_pages_nodemask+0x1e3/0x250
[  333.040557]  [<ffffffff811d6f95>] alloc_pages_vma+0xa5/0x220
[  333.040557]  [<ffffffff811b3514>] handle_mm_fault+0x1034/0x13a0
[  333.040557]  [<ffffffff81061015>] __do_page_fault+0x225/0x4a0
[  333.040557]  [<ffffffff810612b2>] do_page_fault+0x22/0x30
[  333.040557]  [<ffffffff815436b8>] page_fault+0x28/0x30
[  333.118278] Mem-Info:
[  333.120125] active_anon:938017 inactive_anon:12 isolated_anon:0
 active_file:19 inactive_file:37 isolated_file:0
 unevictable:0 dirty:9 writeback:0 unstable:0
 slab_reclaimable:2059 slab_unreclaimable:2379
 mapped:39 shmem:15 pagetables:2536 bounce:0
 free:14966 free_pcp:37 free_cma:0
[  333.164906] Node 0 active_anon:3752068kB inactive_anon:48kB active_file:76kB inactive_file:148kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:156kB dirty:36kB writeback:0kB shmem:60kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
[  333.182929] Node 0 DMA free:15112kB min:184kB low:228kB high:272kB active_anon:788kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  333.202151] lowmem_reserve[]: 0 3732 3732 3732
[  333.207286] Node 0 DMA32 free:44752kB min:44868kB low:56084kB high:67300kB active_anon:3751264kB inactive_anon:48kB active_file:76kB inactive_file:148kB unevictable:0kB writepending:36kB present:3915776kB managed:3839908kB mlocked:0kB slab_reclaimable:8236kB slab_unreclaimable:9516kB kernel_stack:1776kB pagetables:10140kB bounce:0kB free_pcp:148kB local_pcp:148kB free_cma:0kB
[  333.228662] lowmem_reserve[]: 0 0 0 0
[  333.233662] Node 0 DMA: 2*4kB (UM) 2*8kB (UM) 1*16kB (U) 1*32kB (M) 3*64kB (UM) 2*128kB (UM) 1*256kB (U) 0*512kB 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15112kB
[  333.252627] Node 0 DMA32: 84*4kB (UME) 52*8kB (UE) 54*16kB (UME) 44*32kB (UE) 24*64kB (UE) 30*128kB (UME) 6*256kB (UE) 6*512kB (UE) 3*1024kB (U) 4*2048kB (UME) 5*4096kB (U) = 44752kB
[  333.273288] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[  333.279764] 70 total pagecache pages
[  333.282568] 0 pages in swap cache
[  333.285191] Swap cache stats: add 0, delete 0, find 0/0
[  333.288835] Free swap  = 0kB
[  333.291240] Total swap = 0kB
[  333.293614] 982941 pages RAM
[  333.295986] 0 pages HighMem/MovableOnly
[  333.298865] 18988 pages reserved
[  333.301325] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[  333.307110] [ 1576]     0  1576     2732      108      11       3        0         -1000 udevd
[  333.312629] [ 1919]     0  1919    27274       59      21       3        0             0 lvmetad
[  333.318121] [ 1928]     0  1928     6788       48      16       3        0             0 lvmpolld
[  333.324496] [ 2128]     0  2128     2342      121      10       3        0             0 dhclient
[  333.330541] [ 2255]     0  2255     2342      127       9       3        0             0 dhclient
[  333.336664] [ 2297]     0  2297    76516     1442      32       5        0             0 amazon-ssm-agen
[  333.343142] [ 2305]     0  2305    13240       96      25       3        0         -1000 auditd
[  333.349157] [ 2337]     0  2337    61847       83      25       3        0             0 rsyslogd
[  333.355394] [ 2359]     0  2359     1619       26       9       3        0             0 rngd
[  333.360335] [ 2377]    32  2377     8830      105      23       3        0             0 rpcbind
[  333.365483] [ 2398]    29  2398     9972      207      24       3        0             0 rpc.statd
[  333.370496] [ 2429]    81  2429     5450       57      14       3        0             0 dbus-daemon
[  333.376066] [ 2464]     0  2464     1088       41       8       3        0             0 acpid
[  333.380732] [ 2564]     0  2564    20123      207      41       3        0         -1000 sshd
[  333.385566] [ 2574]    38  2574     7443      149      19       3        0             0 ntpd
[  333.390500] [ 2594]     0  2594    22384      430      46       3        0             0 sendmail
[  333.396933] [ 2603]    51  2603    20246      369      41       3        0             0 sendmail
[  333.403230] [ 2615]     0  2615    30402      157      16       3        0             0 crond
[  333.409497] [ 2629]     0  2629     4786       42      14       3        0             0 atd
[  333.415497] [ 2638]     0  2638    29468      272      60       3        0             0 sshd
[  333.421399] [ 2656]     0  2656     1616       37       8       3        0             0 agetty
[  333.427109] [ 2657]     0  2657     1079       31       8       3        0             0 mingetty
[  333.432890] [ 2659]     0  2659     1079       31       7       3        0             0 mingetty
[  333.439658] [ 2661]     0  2661     1079       29       8       3        0             0 mingetty
[  333.445990] [ 2664]     0  2664     1079       30       8       3        0             0 mingetty
[  333.451017] [ 2667]     0  2667     1079       30       8       3        0             0 mingetty
[  333.455913] [ 2669]     0  2669     1079       30       8       3        0             0 mingetty
[  333.463895] [ 2671]     0  2671     2731      101      10       3        0         -1000 udevd
[  333.468825] [ 2672]     0  2672     2731      101      10       3        0         -1000 udevd
[  333.473523] [ 2688]   500  2688    29468      265      59       3        0             0 sshd
[  333.478167] [ 2689]   500  2689    28840      100      15       3        0             0 bash
[  333.483030] [ 2736]   500  2736    31867       70      19       3        0             0 screen
[  333.488037] [ 2737]   500  2737    32002      236      18       3        0             0 screen
[  333.492811] [ 2738]   500  2738    28840      116      14       3        0             0 bash
[  333.497673] [ 2756]   500  2756    28840      110      14       3        0             0 bash
[  333.503036] [ 2785]   500  2785   974688   932682    1839       6        0             0 python
[  333.507934] [ 2786]   500  2786     3796       73      14       3        0             0 top
[  333.512691] Out of memory: Kill process 2785 (python) score 969 or sacrifice child
[  333.519160] Killed process 2785 (python) total-vm:3898752kB, anon-rss:3730620kB, file-rss:108kB, shmem-rss:0kB
[  333.751653] oom_reaper: reaped process 2785 (python), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB