はじめに
以前から話題になっていたMountpoint for Amazon S3が一般公開となったそうなので、使い勝手を確認してみようと思います。
- Mountpoint for Amazon S3 の一般提供を開始
- Mountpoint for Amazon S3 – 一般公開されており、本番環境のワークロードにも対応可能
- GitHub - awslabs/mountpoint-s3
Mountpoint for Amazon S3とは
AWSのS3領域をEC2やECSにマウントして共有ディスクのように使えるアプリケーションです。
S3をマウントして使用できるアプリケーションは今までもs3fsやgoofysなどいくつかありましたが、Mountpoint for Amazon S3はAWSが公式で公開しているのが特徴です。
また、Mountpoint for Amazon S3はファイルシステムをマウントしたような感覚で使用することは出来ますが、POSIX規格には準拠していないため、POSIX規格に準拠したファイル操作と何が異なってくるのかを予め抑えておく必要があります。
Mountpoint for Amazon S3の使用
Mountpoint for Amazon S3は簡単な操作で使用することが出来ます。
以下に実際にS3をマウントして使用できるようにするまでの手順をまとめます。
事前設定
Mountpoint for Amazon S3は、要はクライアントアプリケーションを使ってEC2などのクライアントからS3に対してアクセスするものとなるので、当然クライアントからS3にアクセスするためのIAMユーザ・ロールの権限が必要となります。
以下は該当のS3バケットに対してフルアクセスするためのIAM権限の例です。
IAM権限の例
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "MountpointFullBucketAccess",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::DOC-EXAMPLE-BUCKET"
]
},
{
"Sid": "MountpointFullObjectAccess",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:AbortMultipartUpload",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
]
}
]
}
Mountpoint for Amazon S3のインストール
今回は適当なEC2を用意してS3をマウントしたいと思います。
Mountpoint for Amazon S3自体のインストールは簡単で、RPMパッケージをダウンロードしてyumでインストールするだけです。
公式GitHubにはOSごとに２種類のインストール方法が紹介されております。
Gravitonインスタンスにインストールする場合は、x86_64の部分をarm64に変えると良いそうです。
Mountpoint for Amazon S3のインストール（x86_64版Amazon Linux）
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
sudo yum install -y ./mount-s3.rpm
Mountpoint for Amazon S3のインストール（x86_64版Ubuntu）
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
sudo apt-get install -y ./mount-s3.deb
Mountpoint for Amazon S3のマウント
Mountpoint for Amazon S3をインストールすることで、mount-s3 [S3バケット名] [マウントポイント]の書式でS3をマウントできるようになります。
以下はmountpoint-for-s3-testバケットを、新たに作成したs3mountというマウントポイントにマウントする場合の例。
mount-s3コマンドによるS3マウント
mkdir s3mount
mount-s3 mountpoint-for-s3-test s3mount
以下のようにメッセージが表示されれば正常にマウントできています。
マウント時のメッセージ例
bucket mountpoint-for-s3-test is mounted at s3mount
また、上記のデフォルトマウントコマンドではS3上のオブジェクトを削除できないように制限されているため、削除も行いたい場合は--allow-deleteオプションを付与して以下のようにマウントします。
削除オプションを付与したS3マウント
mount-s3 --allow-delete mountpoint-for-s3-test s3mount
--prefix [プレフィックス名]/を指定することでS3バケット全体ではなく、特定のプレフィックス（フォルダ）部分のみマウントすることもできるので、特定の領域のみユーザに使わせるといったことも可能です。
なお、--prefixオプションを付与する際には最後に「/」を付与するようにしてください。
prefixオプションを付与したS3マウント
mount-s3 --allow-delete --prefix testdir/ mountpoint-for-s3-test s3mount
Mountpoint for Amazon S3のマウント解除
マウント解除は通常のumountで解除します。
S3マウントの解除
umount /s3mount
Mountpoint for Amazon S3の確認
上記手順でS3バケットをマウントすることが出来ましたが、一般的なファイルシステムをマウントした場合と比べてどのような違いがあるのかを、実際にコマンド実行しながら確認していきたいと思います。
マウント時のコマンド確認
共有ディスクをマウントした場合等、dfコマンドを実行するとマウント済みの領域が表示されますが、Mountpoint for Amazon S3でマウントした場合はdfコマンドでは確認できません。
dfコマンドでの確認例
# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 4096 0 4096 0% /dev
tmpfs 486288 0 486288 0% /dev/shm
tmpfs 194516 2904 191612 2% /run
/dev/xvda1 8310764 1640856 6669908 20% /
tmpfs 486288 0 486288 0% /tmp
tmpfs 97256 0 97256 0% /run/user/0
127.0.0.1:/ 9007199254739968 0 9007199254739968 0% /efs
/dev/xvdf 104792064 763656 104028408 1% /io2
mountコマンドで確認することはできるので、マウントできているかを確認する場合はmountコマンドで確認するのが良いかと思います。
mountコマンドで確認した場合の例
# mount
（略）
mountpoint-s3 on /s3mount type fuse (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions)
ファイルリネーム
ファイルリネームには対応していないため実行すると以下のように表示されます。
ファイルリネーム例
# mv testfile testfile.bak
mv: cannot move 'testfile' to 'testfile.bak': Function not implemented
ファイル削除
前述の通り、--allow-deleteを付与しないと以下のように失敗します。
ファイルの削除例
# rm testfile
rm: remove regular empty file 'testfile'? y
rm: cannot remove 'testfile': Operation not permitted
また、--allow-deleteを付与しても、そもそもIAMロールやS3バケットポリシーで削除が許可されていないとできないので注意してください。
ファイル追記
ファイルの追記は以下のように失敗してしまいます。
ファイルの追記例
# echo test >> testfile
-bash: testfile: Operation not permitted
これは通常のS3操作を思い浮かべてもらえればイメージできるかと思いますが、そもそもS3に代表されるオブジェクトストレージは追記を想定したストレージではないのでMountpoint for Amazon S3でも追記することは出来ません。
一応viでの作成・編集は新規ファイルをコピーするような動作となるため出来なくはないようですが、編集する際に以下のようなメッセージが表示され、編集後、スワップファイルが残ってしまうので編集は出来ないものと考えておきましょう。
viでファイル編集した際のメッセージ
E325: ATTENTION
Found a swap file by the name ".testfile.swp"
owned by: root dated: Sat Aug 19 00:19:13 2023
[cannot be read]
While opening file "testfile"
dated: Sat Aug 19 00:19:27 2023
NEWER than swap file!
(1) Another program may be editing the same file. If this is the case,
be careful not to end up with two different instances of the same
file when making changes. Quit, or continue with caution.
(2) An edit session for this file crashed.
If this is the case, use ":recover" or "vim -r testfile"
to recover the changes (see ":help recovery").
If you did this already, delete the swap file ".testfile.swp"
to avoid this message.
Swap file ".testfile.swp" already exists!
[O]pen Read-Only, (E)dit anyway, (R)ecover, (D)elete it, (Q)uit, (A)bort:
ディレクトリの作成
ディレクトリの作成は通常のディレクトリ作成と同様mkdirコマンドで作成することができます。
但し、ディレクトリの中身に何もファイルがない状態の場合、ローカル上での操作となり、S3にアップロードされません。
作成したディレクトリにファイルが格納されることでS3側にアップロードされる動作となるのでご注意ください。
S3領域にディレクトリのみ作成した場合のS3バケットの状態
# mkdir -p /s3mount/testdir1/testdir2
# tree /s3mount
/s3mount
└── testdir1
└── testdir2
2 directories, 0 files
# aws s3 ls s3://mountpoint-for-s3-test/ --recursive
#
作成したディレクトリ配下にファイルを作成した場合のS3バケットの状態
# touch /s3mount/testdir1/testdir2/testfile
# tree /s3mount
/s3mount
└── testdir1
└── testdir2
└── testfile
2 directories, 1 file
# aws s3 ls s3://mountpoint-for-s3-test/ --recursive
2023-08-19 01:47:52 0 testdir/testdir1/testdir2/testfile
リンク作成
lnコマンドで作成できるハードリンク、シンボリックリンクはどちらもサポートされておりません。
シンボリックリンクの作成例
# ln -s /var/log testlink
ln: failed to create symbolic link 'testlink': Operation not permitted
Mountpoint for Amazon S3のベンチマーク
Mountpoint for Amazon S3でマウントした領域はどの程度の性能が出るのか、ベンチマークをしてみようと思います。
ベンチマーク準備
今回はEBS、EFS、Mountpoint for Amazon S3の３つを比較してみようと思います。
以下それぞれのタイプ等。
| ベンチマーク対象 | 設定・タイプ | マウントポイント |
|---|---|---|
| EBS | ボリュームタイプ：プロビジョンドIOPS（io2） サイズ：100GB IOPS：3000 |
/io2 |
| EFS | パフォーマンスモード：汎用 スループットモード：伸縮自在 EFSマウントヘルパー使用 |
/efs |
| Mountpoint for Amazon S3 | ストレージクラス：スタンダード--allow-deleteオプション付与 |
/s3mount |
また、ベンチマークにはAmazon Linux 2023のt3.smallを使用します。
fioの設定
ちょうど「EBSボリュームのベンチマーク」のドキュメントにfioを使用したベンチマーク方法が書かれていたので、こちらを参考にCrystalDiskMark風なパターンでベンチマークしてみようと思います。
ベンチマークで使用したfioの設定は以下となります。
fio.config
[global]
ioengine=libaio
iodepth=1
size=1g
direct=1
runtime=60
directory=/s3mount
filename=.fio-benchmark
norandommap
group_reporting
stonewall
[Seq-Read-1M-Q8-T1]
rw=read
bs=1m
iodepth=8
numjobs=1
[Seq-Write-1M-Q8-T1]
rw=write
bs=1m
iodepth=8
numjobs=1
[Seq-Read-1M-Q1-T1]
rw=read
bs=1m
iodepth=1
numjobs=1
[Seq-Write-1M-Q1-T1]
rw=write
bs=1m
iodepth=1
numjobs=1
[Rand-Read-4K-Q32-T16]
rw=randread
bs=4k
iodepth=32
numjobs=16
[Rand-Write-4K-Q32-T16]
rw=randwrite
bs=4k
iodepth=32
numjobs=16
[Rand-Read-4K-Q1-T1]
rw=randread
bs=4k
iodepth=1
numjobs=1
[Rand-Write-4K-Q1-T1]
rw=randwrite
bs=4k
iodepth=1
numjobs=1
directory部分をEBSやEFS、Mountpoint for Amazon S3をマウントしたパスに書き換えて以下のように実行します。
fioの実行
fio fio.config
ベンチマーク実行結果
ベンチマークを行った結果が以下となります。
Mountpoint for Amazon S3のWriteの性能を「ー」としておりますが、上述の通り追記等の処理が行えないためfioを実行してもエラーとなるためです。
EBSのランダムRead・WriteはIOPSの上限に引っかかっているかと思うので、実際はもっと性能が出るかと思いますが、今回はMountpoint for Amazon S3の評価が本題なのでこのままとします。
| パターン | EBS転送速度 | EBS IOPS | EFS転送速度 | EFS IOPS | S3転送速度 | S3 IOPS |
|---|---|---|---|---|---|---|
| シーケンシャルRead 1M Q8 T1 |
344MB/s | 327 | 213MB/s | 203 | 260MB/s | 247 |
| シーケンシャルWrite 1M Q8 T1 |
280MB/s | 267 | 144MB/s | 137 | ー | ー |
| シーケンシャルRead 1M Q1 T1 |
286MB/s | 272 | 47.2MB/s | 45 | 296MB/s | 282 |
| シーケンシャルWrite 1M Q1 T1 |
285MB/s | 272 | 34.9MB/s | 33 | ー | ー |
| ランダムRead 4K Q32 T16 |
12.5MB/s | 3049 | 43.7MB/s | 10.7k | 382kB/s | 93 |
| ランダムWrite 4K Q32 T16 |
12.3MB/s | 3015 | 19.3MB/s | 4702 | ー | ー |
| ランダムRead 4K Q1 T1 |
12.4MB/s | 3019 | 1241kB/s | 302 | 51.3kB/s | 12 |
| ランダムWrite 4K Q1 T1 |
12.3MB/s | 3014 | 442kB/s | 107 | ー | ー |
各パターンの詳細を見たい方は以下展開して確認してください。
EBS結果（展開してください）
EBSのfio結果
Seq-Read-1M-Q8-T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Write-1M-Q8-T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Read-1M-Q1-T1: (g=2): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seq-Write-1M-Q1-T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Rand-Read-4K-Q32-T16: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Write-4K-Q32-T16: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Read-4K-Q1-T1: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Rand-Write-4K-Q1-T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32
Starting 38 processes
Jobs: 1 (f=0): [_(37),f(1)][100.0%][w=11.0MiB/s][w=2804 IOPS][eta 00m:00s]
Seq-Read-1M-Q8-T1: (groupid=0, jobs=1): err= 0: pid=156247: Sun Aug 20 02:14:52 2023
read: IOPS=327, BW=328MiB/s (344MB/s)(1024MiB/3122msec)
slat (usec): min=23, max=6936, avg=2013.71, stdev=1536.82
clat (usec): min=2149, max=53106, avg=22330.75, stdev=11665.09
lat (usec): min=2180, max=56222, avg=24344.46, stdev=12859.52
clat percentiles (usec):
| 1.00th=[ 2573], 5.00th=[ 3032], 10.00th=[ 3458], 20.00th=[ 3949],
| 30.00th=[28443], 40.00th=[28705], 50.00th=[28967], 60.00th=[28967],
| 70.00th=[29230], 80.00th=[29230], 90.00th=[30016], 95.00th=[32113],
| 99.00th=[33162], 99.50th=[36963], 99.90th=[49021], 99.95th=[53216],
| 99.99th=[53216]
bw ( KiB/s): min=253952, max=647739, per=99.21%, avg=333220.60, stdev=175823.13, samples=5
iops : min= 248, max= 632, avg=325.20, stdev=171.51, samples=5
lat (msec) : 4=20.80%, 10=6.74%, 20=0.29%, 50=72.07%, 100=0.10%
cpu : usr=0.54%, sys=1.86%, ctx=1651, majf=0, minf=2059
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Seq-Write-1M-Q8-T1: (groupid=1, jobs=1): err= 0: pid=156251: Sun Aug 20 02:14:52 2023
write: IOPS=267, BW=267MiB/s (280MB/s)(1024MiB/3830msec); 0 zone resets
slat (usec): min=44, max=16322, avg=2231.71, stdev=1718.08
clat (usec): min=1655, max=53202, avg=27652.55, stdev=8100.48
lat (usec): min=1732, max=56471, avg=29884.27, stdev=8665.53
clat percentiles (usec):
| 1.00th=[ 3163], 5.00th=[ 3458], 10.00th=[21890], 20.00th=[28705],
| 30.00th=[28967], 40.00th=[28967], 50.00th=[28967], 60.00th=[28967],
| 70.00th=[28967], 80.00th=[31589], 90.00th=[32113], 95.00th=[34866],
| 99.00th=[46400], 99.50th=[48497], 99.90th=[52691], 99.95th=[53216],
| 99.99th=[53216]
bw ( KiB/s): min=253952, max=354837, per=98.29%, avg=269094.43, stdev=37818.95, samples=7
iops : min= 248, max= 346, avg=262.57, stdev=36.80, samples=7
lat (msec) : 2=0.10%, 4=7.81%, 10=0.39%, 20=1.46%, 50=89.94%
lat (msec) : 100=0.29%
cpu : usr=1.88%, sys=1.41%, ctx=1678, majf=0, minf=12
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Seq-Read-1M-Q1-T1: (groupid=2, jobs=1): err= 0: pid=156257: Sun Aug 20 02:14:52 2023
read: IOPS=272, BW=273MiB/s (286MB/s)(1024MiB/3751msec)
slat (usec): min=20, max=4781, avg=47.30, stdev=152.79
clat (usec): min=684, max=7294, avg=3609.84, stdev=1057.06
lat (usec): min=746, max=7328, avg=3657.14, stdev=1058.08
clat percentiles (usec):
| 1.00th=[ 750], 5.00th=[ 791], 10.00th=[ 865], 20.00th=[ 3916],
| 30.00th=[ 3949], 40.00th=[ 3982], 50.00th=[ 3982], 60.00th=[ 3982],
| 70.00th=[ 3982], 80.00th=[ 4015], 90.00th=[ 4015], 95.00th=[ 4047],
| 99.00th=[ 5014], 99.50th=[ 5669], 99.90th=[ 7242], 99.95th=[ 7308],
| 99.99th=[ 7308]
bw ( KiB/s): min=253952, max=428740, per=100.00%, avg=279652.00, stdev=65747.39, samples=7
iops : min= 248, max= 418, avg=272.86, stdev=64.01, samples=7
lat (usec) : 750=0.68%, 1000=9.96%
lat (msec) : 2=1.27%, 4=68.75%, 10=19.34%
cpu : usr=0.88%, sys=0.91%, ctx=1025, majf=0, minf=266
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Seq-Write-1M-Q1-T1: (groupid=3, jobs=1): err= 0: pid=156262: Sun Aug 20 02:14:52 2023
write: IOPS=272, BW=272MiB/s (285MB/s)(1024MiB/3764msec); 0 zone resets
slat (usec): min=42, max=1731, avg=105.84, stdev=76.87
clat (usec): min=1130, max=8655, avg=3564.31, stdev=965.57
lat (usec): min=1172, max=8732, avg=3670.15, stdev=972.52
clat percentiles (usec):
| 1.00th=[ 1156], 5.00th=[ 1205], 10.00th=[ 1270], 20.00th=[ 3818],
| 30.00th=[ 3884], 40.00th=[ 3884], 50.00th=[ 3916], 60.00th=[ 3916],
| 70.00th=[ 3949], 80.00th=[ 3949], 90.00th=[ 3982], 95.00th=[ 4015],
| 99.00th=[ 4883], 99.50th=[ 5800], 99.90th=[ 6849], 99.95th=[ 8717],
| 99.99th=[ 8717]
bw ( KiB/s): min=253952, max=387786, per=98.28%, avg=273801.43, stdev=50270.01, samples=7
iops : min= 248, max= 378, avg=267.14, stdev=48.89, samples=7
lat (msec) : 2=13.38%, 4=80.18%, 10=6.45%
cpu : usr=2.02%, sys=0.93%, ctx=1025, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Rand-Read-4K-Q32-T16: (groupid=4, jobs=16): err= 0: pid=156267: Sun Aug 20 02:14:52 2023
read: IOPS=3049, BW=11.9MiB/s (12.5MB/s)(715MiB/60027msec)
slat (nsec): min=1924, max=128039k, avg=5238465.70, stdev=9531041.06
clat (usec): min=181, max=474989, avg=162614.04, stdev=51900.61
lat (usec): min=189, max=475000, avg=167852.51, stdev=52879.82
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 80], 10.00th=[ 104], 20.00th=[ 126],
| 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 176],
| 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 226], 95.00th=[ 247],
| 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 342], 99.95th=[ 363],
| 99.99th=[ 426]
bw ( KiB/s): min= 7305, max=31279, per=99.67%, avg=12159.87, stdev=168.93, samples=1904
iops : min= 1826, max= 7814, avg=3037.86, stdev=42.21, samples=1904
lat (usec) : 250=0.04%, 500=0.10%, 750=0.03%, 1000=0.03%
lat (msec) : 2=0.20%, 4=0.62%, 10=0.45%, 20=0.27%, 50=0.92%
lat (msec) : 100=6.51%, 250=86.52%, 500=4.31%
cpu : usr=0.12%, sys=0.20%, ctx=135988, majf=0, minf=677
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.7%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=183078,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Rand-Write-4K-Q32-T16: (groupid=5, jobs=16): err= 0: pid=156404: Sun Aug 20 02:14:52 2023
write: IOPS=3015, BW=11.8MiB/s (12.3MB/s)(707MiB/60027msec); 0 zone resets
slat (usec): min=2, max=296642, avg=5298.40, stdev=15383.26
clat (usec): min=138, max=590686, avg=164448.26, stdev=71623.19
lat (usec): min=242, max=590689, avg=169746.66, stdev=72696.41
clat percentiles (msec):
| 1.00th=[ 44], 5.00th=[ 73], 10.00th=[ 86], 20.00th=[ 104],
| 30.00th=[ 120], 40.00th=[ 136], 50.00th=[ 153], 60.00th=[ 171],
| 70.00th=[ 194], 80.00th=[ 220], 90.00th=[ 259], 95.00th=[ 296],
| 99.00th=[ 380], 99.50th=[ 409], 99.90th=[ 481], 99.95th=[ 493],
| 99.99th=[ 542]
bw ( KiB/s): min= 5264, max=20646, per=99.82%, avg=12038.22, stdev=172.27, samples=1897
iops : min= 1316, max= 5156, avg=3007.51, stdev=43.04, samples=1897
lat (usec) : 250=0.01%, 500=0.01%, 750=0.05%, 1000=0.09%
lat (msec) : 2=0.07%, 4=0.02%, 10=0.08%, 20=0.05%, 50=1.03%
lat (msec) : 100=16.45%, 250=70.06%, 500=12.04%, 750=0.03%
cpu : usr=0.10%, sys=0.18%, ctx=162798, majf=0, minf=159
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.7%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,180986,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Rand-Read-4K-Q1-T1: (groupid=6, jobs=1): err= 0: pid=156539: Sun Aug 20 02:14:52 2023
read: IOPS=3019, BW=11.8MiB/s (12.4MB/s)(708MiB/60001msec)
slat (usec): min=2, max=12040, avg= 6.52, stdev=46.70
clat (nsec): min=1999, max=15429k, avg=322639.92, stdev=216497.01
lat (usec): min=89, max=15432, avg=329.16, stdev=220.85
clat percentiles (usec):
| 1.00th=[ 102], 5.00th=[ 113], 10.00th=[ 123], 20.00th=[ 161],
| 30.00th=[ 297], 40.00th=[ 314], 50.00th=[ 322], 60.00th=[ 330],
| 70.00th=[ 338], 80.00th=[ 355], 90.00th=[ 506], 95.00th=[ 537],
| 99.00th=[ 889], 99.50th=[ 1221], 99.90th=[ 2835], 99.95th=[ 3949],
| 99.99th=[ 6325]
bw ( KiB/s): min=11823, max=19900, per=99.97%, avg=12075.57, stdev=727.08, samples=118
iops : min= 2955, max= 4975, avg=3018.85, stdev=181.78, samples=118
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.64%, 250=23.47%, 500=64.79%, 750=9.51%, 1000=0.79%
lat (msec) : 2=0.58%, 4=0.16%, 10=0.04%, 20=0.01%
cpu : usr=1.45%, sys=2.80%, ctx=181235, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=181188,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Rand-Write-4K-Q1-T1: (groupid=7, jobs=1): err= 0: pid=156603: Sun Aug 20 02:14:52 2023
write: IOPS=3014, BW=11.8MiB/s (12.3MB/s)(707MiB/60001msec); 0 zone resets
slat (usec): min=2, max=7151, avg= 7.62, stdev=40.18
clat (usec): min=2, max=9981, avg=321.91, stdev=193.70
lat (usec): min=177, max=9987, avg=329.54, stdev=197.56
clat percentiles (usec):
| 1.00th=[ 190], 5.00th=[ 200], 10.00th=[ 210], 20.00th=[ 258],
| 30.00th=[ 302], 40.00th=[ 314], 50.00th=[ 322], 60.00th=[ 326],
| 70.00th=[ 334], 80.00th=[ 343], 90.00th=[ 355], 95.00th=[ 392],
| 99.00th=[ 766], 99.50th=[ 1254], 99.90th=[ 3359], 99.95th=[ 4113],
| 99.99th=[ 5997]
bw ( KiB/s): min=11512, max=17744, per=100.00%, avg=12071.06, stdev=547.70, samples=119
iops : min= 2878, max= 4436, avg=3017.76, stdev=136.93, samples=119
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=19.19%, 500=78.69%, 750=1.07%, 1000=0.35%
lat (msec) : 2=0.42%, 4=0.20%, 10=0.06%
cpu : usr=1.38%, sys=2.23%, ctx=181109, majf=0, minf=9
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,180898,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=1024MiB (1074MB), run=3122-3122msec
Run status group 1 (all jobs):
WRITE: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=1024MiB (1074MB), run=3830-3830msec
Run status group 2 (all jobs):
READ: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=1024MiB (1074MB), run=3751-3751msec
Run status group 3 (all jobs):
WRITE: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=1024MiB (1074MB), run=3764-3764msec
Run status group 4 (all jobs):
READ: bw=11.9MiB/s (12.5MB/s), 11.9MiB/s-11.9MiB/s (12.5MB/s-12.5MB/s), io=715MiB (750MB), run=60027-60027msec
Run status group 5 (all jobs):
WRITE: bw=11.8MiB/s (12.3MB/s), 11.8MiB/s-11.8MiB/s (12.3MB/s-12.3MB/s), io=707MiB (741MB), run=60027-60027msec
Run status group 6 (all jobs):
READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=708MiB (742MB), run=60001-60001msec
Run status group 7 (all jobs):
WRITE: bw=11.8MiB/s (12.3MB/s), 11.8MiB/s-11.8MiB/s (12.3MB/s-12.3MB/s), io=707MiB (741MB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=372458/369537, merge=0/0, ticks=3817558/3740979, in_queue=7558537, util=99.15%
EFS結果（展開してください）
EFSのfio結果
Seq-Read-1M-Q8-T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Write-1M-Q8-T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Read-1M-Q1-T1: (g=2): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seq-Write-1M-Q1-T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Rand-Read-4K-Q32-T16: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Write-4K-Q32-T16: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Read-4K-Q1-T1: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Rand-Write-4K-Q1-T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32
Starting 38 processes
Seq-Read-1M-Q8-T1: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [_(37),w(1)][13.1%][w=440KiB/s][w=110 IOPS][eta 34m:04s]
Seq-Read-1M-Q8-T1: (groupid=0, jobs=1): err= 0: pid=165964: Sun Aug 20 03:56:48 2023
read: IOPS=203, BW=203MiB/s (213MB/s)(1024MiB/5037msec)
slat (usec): min=49, max=2487, avg=139.14, stdev=149.19
clat (msec): min=16, max=236, avg=39.04, stdev=24.60
lat (msec): min=16, max=236, avg=39.18, stdev=24.60
clat percentiles (msec):
| 1.00th=[ 20], 5.00th=[ 22], 10.00th=[ 23], 20.00th=[ 26],
| 30.00th=[ 29], 40.00th=[ 31], 50.00th=[ 33], 60.00th=[ 35],
| 70.00th=[ 39], 80.00th=[ 44], 90.00th=[ 56], 95.00th=[ 91],
| 99.00th=[ 148], 99.50th=[ 163], 99.90th=[ 228], 99.95th=[ 236],
| 99.99th=[ 236]
bw ( KiB/s): min=175028, max=235991, per=100.00%, avg=209907.00, stdev=18602.14, samples=9
iops : min= 170, max= 230, avg=204.67, stdev=18.25, samples=9
lat (msec) : 20=2.64%, 50=84.86%, 100=8.11%, 250=4.39%
cpu : usr=0.10%, sys=1.89%, ctx=1552, majf=0, minf=2058
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Seq-Write-1M-Q8-T1: (groupid=1, jobs=1): err= 0: pid=165970: Sun Aug 20 03:56:48 2023
write: IOPS=137, BW=137MiB/s (144MB/s)(1024MiB/7450msec); 0 zone resets
slat (usec): min=91, max=3787, avg=186.43, stdev=202.13
clat (msec): min=20, max=302, avg=57.91, stdev=29.14
lat (msec): min=21, max=302, avg=58.09, stdev=29.14
clat percentiles (msec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 32], 20.00th=[ 37],
| 30.00th=[ 45], 40.00th=[ 51], 50.00th=[ 56], 60.00th=[ 61],
| 70.00th=[ 66], 80.00th=[ 71], 90.00th=[ 79], 95.00th=[ 87],
| 99.00th=[ 186], 99.50th=[ 241], 99.90th=[ 288], 99.95th=[ 305],
| 99.99th=[ 305]
bw ( KiB/s): min=126470, max=151552, per=99.97%, avg=140701.07, stdev=8646.52, samples=14
iops : min= 123, max= 148, avg=137.29, stdev= 8.48, samples=14
lat (msec) : 50=39.36%, 100=57.42%, 250=2.73%, 500=0.49%
cpu : usr=0.78%, sys=1.53%, ctx=1046, majf=0, minf=10
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Seq-Read-1M-Q1-T1: (groupid=2, jobs=1): err= 0: pid=165979: Sun Aug 20 03:56:48 2023
read: IOPS=45, BW=45.0MiB/s (47.2MB/s)(1024MiB/22740msec)
slat (usec): min=61, max=7907, avg=100.47, stdev=300.69
clat (usec): min=13306, max=97495, avg=22095.31, stdev=7013.49
lat (usec): min=13782, max=97571, avg=22195.78, stdev=7025.72
clat percentiles (usec):
| 1.00th=[16581], 5.00th=[17433], 10.00th=[17957], 20.00th=[18482],
| 30.00th=[19006], 40.00th=[19530], 50.00th=[20055], 60.00th=[20841],
| 70.00th=[21890], 80.00th=[23987], 90.00th=[27657], 95.00th=[31589],
| 99.00th=[53740], 99.50th=[61080], 99.90th=[95945], 99.95th=[96994],
| 99.99th=[96994]
bw ( KiB/s): min=28500, max=53354, per=99.86%, avg=46045.75, stdev=5430.42, samples=44
iops : min= 27, max= 52, avg=44.84, stdev= 5.40, samples=44
lat (msec) : 20=47.36%, 50=51.07%, 100=1.56%
cpu : usr=0.05%, sys=0.38%, ctx=1044, majf=0, minf=265
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Seq-Write-1M-Q1-T1: (groupid=3, jobs=1): err= 0: pid=166006: Sun Aug 20 03:56:48 2023
write: IOPS=33, BW=33.3MiB/s (34.9MB/s)(1024MiB/30761msec); 0 zone resets
slat (usec): min=102, max=3276, avg=179.53, stdev=153.81
clat (msec): min=20, max=422, avg=29.85, stdev=18.51
lat (msec): min=21, max=423, avg=30.03, stdev=18.51
clat percentiles (msec):
| 1.00th=[ 24], 5.00th=[ 25], 10.00th=[ 26], 20.00th=[ 26],
| 30.00th=[ 27], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 29],
| 70.00th=[ 30], 80.00th=[ 31], 90.00th=[ 34], 95.00th=[ 36],
| 99.00th=[ 48], 99.50th=[ 113], 99.90th=[ 359], 99.95th=[ 422],
| 99.99th=[ 422]
bw ( KiB/s): min= 6144, max=38989, per=99.77%, avg=34011.23, stdev=5932.22, samples=60
iops : min= 6, max= 38, avg=33.12, stdev= 5.81, samples=60
lat (msec) : 50=99.02%, 100=0.39%, 250=0.39%, 500=0.20%
cpu : usr=0.29%, sys=0.29%, ctx=1033, majf=0, minf=9
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Rand-Read-4K-Q32-T16: (groupid=4, jobs=16): err= 0: pid=166042: Sun Aug 20 03:56:48 2023
read: IOPS=10.7k, BW=41.7MiB/s (43.7MB/s)(2504MiB/60037msec)
slat (nsec): min=1881, max=9893.2k, avg=23118.50, stdev=73466.89
clat (msec): min=4, max=178, avg=47.92, stdev= 9.80
lat (msec): min=4, max=178, avg=47.94, stdev= 9.80
clat percentiles (msec):
| 1.00th=[ 33], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41],
| 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 48],
| 70.00th=[ 51], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 65],
| 99.00th=[ 79], 99.50th=[ 87], 99.90th=[ 146], 99.95th=[ 157],
| 99.99th=[ 167]
bw ( KiB/s): min=24408, max=55360, per=100.00%, avg=42738.87, stdev=325.55, samples=1904
iops : min= 6090, max=13840, avg=10682.66, stdev=81.36, samples=1904
lat (msec) : 10=0.01%, 20=0.02%, 50=67.94%, 100=31.83%, 250=0.21%
cpu : usr=0.38%, sys=0.94%, ctx=1035226, majf=0, minf=652
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=641041,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Rand-Write-4K-Q32-T16: (groupid=5, jobs=16): err= 0: pid=166176: Sun Aug 20 03:56:48 2023
write: IOPS=4702, BW=18.4MiB/s (19.3MB/s)(1104MiB/60099msec); 0 zone resets
slat (usec): min=2, max=8942, avg=32.17, stdev=100.44
clat (msec): min=10, max=1307, avg=108.81, stdev=52.17
lat (msec): min=10, max=1307, avg=108.84, stdev=52.17
clat percentiles (msec):
| 1.00th=[ 94], 5.00th=[ 96], 10.00th=[ 97], 20.00th=[ 100],
| 30.00th=[ 101], 40.00th=[ 103], 50.00th=[ 104], 60.00th=[ 105],
| 70.00th=[ 107], 80.00th=[ 110], 90.00th=[ 115], 95.00th=[ 121],
| 99.00th=[ 161], 99.50th=[ 634], 99.90th=[ 953], 99.95th=[ 961],
| 99.99th=[ 961]
bw ( KiB/s): min= 6752, max=20938, per=100.00%, avg=19142.83, stdev=125.47, samples=1856
iops : min= 1688, max= 5234, avg=4782.65, stdev=31.37, samples=1856
lat (msec) : 20=0.01%, 50=0.05%, 100=26.98%, 250=72.25%, 500=0.16%
lat (msec) : 750=0.22%, 1000=0.33%, 2000=0.01%
cpu : usr=0.20%, sys=0.39%, ctx=523481, majf=0, minf=151
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,282635,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Rand-Read-4K-Q1-T1: (groupid=6, jobs=1): err= 0: pid=166311: Sun Aug 20 03:56:48 2023
read: IOPS=302, BW=1212KiB/s (1241kB/s)(71.0MiB/60002msec)
slat (usec): min=8, max=3208, avg=17.99, stdev=56.87
clat (usec): min=594, max=29700, avg=3274.09, stdev=1037.00
lat (usec): min=606, max=29721, avg=3292.08, stdev=1040.45
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 2507], 10.00th=[ 2638], 20.00th=[ 2769],
| 30.00th=[ 2868], 40.00th=[ 2966], 50.00th=[ 3064], 60.00th=[ 3261],
| 70.00th=[ 3458], 80.00th=[ 3687], 90.00th=[ 4228], 95.00th=[ 5145],
| 99.00th=[ 6915], 99.50th=[ 8094], 99.90th=[10683], 99.95th=[11731],
| 99.99th=[16909]
bw ( KiB/s): min= 1029, max= 1376, per=100.00%, avg=1212.30, stdev=63.28, samples=119
iops : min= 257, max= 344, avg=302.95, stdev=15.85, samples=119
lat (usec) : 750=2.94%, 1000=0.69%
lat (msec) : 2=0.16%, 4=83.86%, 10=12.19%, 20=0.16%, 50=0.01%
cpu : usr=0.35%, sys=0.53%, ctx=18381, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=18180,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Rand-Write-4K-Q1-T1: (groupid=7, jobs=1): err= 0: pid=166374: Sun Aug 20 03:56:48 2023
write: IOPS=107, BW=432KiB/s (442kB/s)(25.3MiB/60008msec); 0 zone resets
slat (usec): min=13, max=1871, avg=23.89, stdev=38.04
clat (usec): min=6872, max=74459, avg=9229.58, stdev=1743.69
lat (usec): min=6941, max=74477, avg=9253.46, stdev=1744.96
clat percentiles (usec):
| 1.00th=[ 7373], 5.00th=[ 7635], 10.00th=[ 7832], 20.00th=[ 8094],
| 30.00th=[ 8291], 40.00th=[ 8586], 50.00th=[ 8848], 60.00th=[ 9241],
| 70.00th=[ 9634], 80.00th=[10028], 90.00th=[10945], 95.00th=[12387],
| 99.00th=[14484], 99.50th=[15795], 99.90th=[20579], 99.95th=[24773],
| 99.99th=[74974]
bw ( KiB/s): min= 368, max= 456, per=100.00%, avg=432.16, stdev=13.20, samples=119
iops : min= 92, max= 114, avg=108.02, stdev= 3.30, samples=119
lat (msec) : 10=79.13%, 20=20.72%, 50=0.14%, 100=0.02%
cpu : usr=0.24%, sys=0.22%, ctx=6553, majf=0, minf=8
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6478,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=203MiB/s (213MB/s), 203MiB/s-203MiB/s (213MB/s-213MB/s), io=1024MiB (1074MB), run=5037-5037msec
Run status group 1 (all jobs):
WRITE: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=1024MiB (1074MB), run=7450-7450msec
Run status group 2 (all jobs):
READ: bw=45.0MiB/s (47.2MB/s), 45.0MiB/s-45.0MiB/s (47.2MB/s-47.2MB/s), io=1024MiB (1074MB), run=22740-22740msec
Run status group 3 (all jobs):
WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=1024MiB (1074MB), run=30761-30761msec
Run status group 4 (all jobs):
READ: bw=41.7MiB/s (43.7MB/s), 41.7MiB/s-41.7MiB/s (43.7MB/s-43.7MB/s), io=2504MiB (2626MB), run=60037-60037msec
Run status group 5 (all jobs):
WRITE: bw=18.4MiB/s (19.3MB/s), 18.4MiB/s-18.4MiB/s (19.3MB/s-19.3MB/s), io=1104MiB (1158MB), run=60099-60099msec
Run status group 6 (all jobs):
READ: bw=1212KiB/s (1241kB/s), 1212KiB/s-1212KiB/s (1241kB/s-1241kB/s), io=71.0MiB (74.5MB), run=60002-60002msec
Run status group 7 (all jobs):
WRITE: bw=432KiB/s (442kB/s), 432KiB/s-432KiB/s (442kB/s-442kB/s), io=25.3MiB (26.5MB), run=60008-60008msec
S3結果（展開してください）
S3のfio結果
Seq-Read-1M-Q8-T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Write-1M-Q8-T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Read-1M-Q1-T1: (g=2): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seq-Write-1M-Q1-T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Rand-Read-4K-Q32-T16: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Write-4K-Q32-T16: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Read-4K-Q1-T1: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Rand-Write-4K-Q1-T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32
Starting 38 processes
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=0, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1048576, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=2097152, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=3145728, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=4194304, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=5242880, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=6291456, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=7340032, buflen=1048576
fio: pid=160247, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=0, buflen=1048576
fio: pid=160253, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=935215104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=857464832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=468156416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=985276416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=8654848, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1052733440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=18911232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=119975936, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=176386048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=547172352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=81301504, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=459362304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=710455296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=295763968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1033744384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=117538816, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=495435776, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=669491200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=294543360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=384147456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=914653184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=477323264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1014996992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=792219648, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=912949248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=542818304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=746360832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=936366080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=251334656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=784572416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=70127616, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=518848512, buflen=4096
fio: pid=160333, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=263307264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=615055360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=532480000, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=384966656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=421335040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=61181952, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=708857856, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=121028608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=313151488, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=969424896, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=824680448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=780730368, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=976297984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=478814208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=816156672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=89661440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=14708736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=16232448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=305840128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=32333824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=642404352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=538820608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=858578944, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=930562048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=578523136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=753684480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1038864384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=864620544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=504500224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=402882560, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=838029312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=954761216, buflen=4096
fio: pid=160338, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=523722752, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=542588928, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=308756480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=268566528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=293478400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=146636800, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=371425280, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=955932672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=382050304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=310792192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=775139328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=548966400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=71241728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=941813760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=483074048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=798691328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=669417472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=886464512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12976128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=669818880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=694996992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=276897792, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=837181440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=102940672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=76701696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=840101888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=243916800, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=145715200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=674836480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=743108608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=661725184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=438169600, buflen=4096
fio: pid=160335, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=934752256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=99217408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=425132032, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=806121472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=990658560, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=346288128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=953012224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=28512256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=499052544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=478146560, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=823971840, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=776744960, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=754278400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=19656704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=429019136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=959897600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12582912, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=697966592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12124160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=813838336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=506232832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1027772416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=543240192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=220508160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=342061056, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=364220416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=304779264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=954494976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=445227008, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=58896384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=610508800, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=981893120, buflen=4096
fio: pid=160348, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=8601600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=878317568, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=252477440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=479391744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=311762944, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=968626176, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=132915200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=731619328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=813502464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=238047232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=303321088, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=870821888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=412164096, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=488108032, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=219828224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=409120768, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=645390336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=716910592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=960286720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=401412096, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=823185408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=435511296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1062674432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=922300416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=36704256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=166371328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1007411200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=102277120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=385798144, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=764690432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=281698304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1055588352, buflen=4096
fio: pid=160346, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=941694976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=658329600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=696217600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=885084160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=448086016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=755429376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=52625408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1004957696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=514899968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=583254016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=663240704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=23285760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=765550592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=912629760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=719757312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=533606400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=888299520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=831922176, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=581382144, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=699023360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1036546048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=37203968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=428335104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1067794432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=894111744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=179408896, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1041235968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=395902976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1019281408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=923275264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=131219456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=359477248, buflen=4096
fio: pid=160342, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=239673344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=440659968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=485097472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=549363712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=230699008, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=588984320, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=921407488, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=4083712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=535695360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=778104832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=181014528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=579887104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=976420864, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=194121728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1031229440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=739954688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=357748736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=407511040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=61419520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=941998080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=712503296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=996265984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=494002176, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=562319360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=600989696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=924049408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12562432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=700940288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=345415680, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=171368448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=916762624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=29659136, buflen=4096
fio: pid=160340, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=977674240, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=284614656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=840081408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=615800832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=946143232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=839675904, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=585740288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=488005632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=976441344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=573325312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1002610688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=660320256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=390197248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=315158528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=652472320, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=578666496, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=108425216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=151556096, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=785080320, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=466202624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=653971456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=129118208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=771239936, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=678895616, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=581734400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=824451072, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=615186432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030008832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=324722688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=232640512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=775360512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=440455168, buflen=4096
fio: pid=160339, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=468844544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=816947200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=331497472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030017024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=289689600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=223203328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=801005568, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=518500352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=688750592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=674770944, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=907816960, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1002545152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=44814336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=502349824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=216772608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=322326528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=959201280, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=774053888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=942166016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=149180416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=766337024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=159719424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=511873024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=492613632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=124293120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=967725056, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=829657088, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030090752, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=934739968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=275898368, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=248991744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=373313536, buflen=4096
fio: pid=160343, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=249245696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=618217472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=426070016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=714809344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=770711552, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=135536640, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=119123968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=611643392, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=596529152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=678940672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=642818048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=351408128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=625250304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=970063872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=596107264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=657723392, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=168054784, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=89559040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=149348352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=411435008, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=879968256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1021427712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=288948224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=204644352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=219381760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=814739456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=11239424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=979767296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=127823872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=99774464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=260157440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=109899776, buflen=4096
fio: pid=160334, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=628183040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=213262336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=919969792, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=970616832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=883093504, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1224704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=534097920, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=638631936, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=908935168, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=413306880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=901947392, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=478990336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=831348736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=488243200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1055608832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030279168, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1061765120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=853839872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=117583872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1066917888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=160763904, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=939462656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=317804544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=954675200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=695492608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=305479680, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=670142464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=855990272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=376033280, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=180854784, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=764383232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=213557248, buflen=4096
fio: pid=160344, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=171302912, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=426074112, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=631111680, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1049264128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=768012288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=506085376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=127107072, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=943030272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=178184192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=188891136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=14569472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=491732992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=330113024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=398356480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=35921920, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=266612736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=580145152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=316366848, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=762724352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=827146240, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=547999744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=52604928, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=744288256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=389206016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=780058624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=138600448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=568922112, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=947412992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=863789056, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=861249536, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=100061184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=852398080, buflen=4096
fio: pid=160337, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=418398208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=45109248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=866889728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=860798976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=206725120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=248606720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1016901632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=734695424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=123371520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=401780736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=597393408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=365465600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=230748160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=632897536, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=481193984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=670666752, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=588877824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=727089152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=951537664, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=538853376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=309227520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=531943424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=871272448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=495022080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=853266432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=253792256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=360841216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=251990016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=970469376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=50356224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=813268992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=242343936, buflen=4096
fio: pid=160336, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=693460992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=430518272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=229711872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=563814400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=193257472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=79380480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=333225984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=412213248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=33120256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=297811968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=733782016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=771977216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1027985408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=68124672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=781545472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=998039552, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1013321728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1022607360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=184078336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=568000512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=452157440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=175161344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=642146304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=682221568, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=571371520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=67141632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=37384192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=789626880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=604008448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=536186880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1025413120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=72269824, buflen=4096
fio: pid=160345, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=190599168, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=663441408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=490049536, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=794877952, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=233885696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1005789184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1011388416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=909500416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1038233600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=777646080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=626528256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=735145984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=676507648, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=982581248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1060880384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=388255744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=814243840, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=420757504, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=627343360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=442470400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=361467904, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=964497408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=576286720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=581394432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=28442624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=78434304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=784994304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1011179520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=102154240, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=189636608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=48988160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=521666560, buflen=4096
fio: pid=160347, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=455966720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=982298624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=189042688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=827351040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=582787072, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=80031744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=105869312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=555749376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=155660288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1028173824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=651608064, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=94019584, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=562163712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1055100928, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=803667968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=906334208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=426459136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=154656768, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=415268864, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=503599104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=142069760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=313790464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=770019328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=208777216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=874934272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=432349184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=153452544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=909135872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=746696704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=912879616, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=123228160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=411148288, buflen=4096
fio: pid=160341, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=251985920, buflen=4096
fio: pid=160467, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
Seq-Read-1M-Q8-T1: (groupid=0, jobs=1): err= 0: pid=160241: Sun Aug 20 02:54:25 2023
read: IOPS=247, BW=248MiB/s (260MB/s)(1024MiB/4135msec)
slat (usec): min=426, max=328857, avg=3972.90, stdev=22596.07
clat (usec): min=4, max=335599, avg=25631.80, stdev=53986.45
lat (usec): min=453, max=517739, avg=29604.70, stdev=59102.60
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6],
| 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9],
| 70.00th=[ 13], 80.00th=[ 28], 90.00th=[ 59], 95.00th=[ 90],
| 99.00th=[ 330], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 338],
| 99.99th=[ 338]
bw ( KiB/s): min= 1793, max=372736, per=96.84%, avg=245570.62, stdev=129791.86, samples=8
iops : min= 1, max= 364, avg=239.62, stdev=126.95, samples=8
lat (usec) : 10=0.10%, 500=0.10%
lat (msec) : 2=0.20%, 4=1.27%, 10=62.60%, 20=13.77%, 50=7.13%
lat (msec) : 100=10.64%, 250=2.15%, 500=2.05%
cpu : usr=0.60%, sys=1.33%, ctx=1029, majf=0, minf=2059
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Seq-Write-1M-Q8-T1: (groupid=1, jobs=1): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160247: Sun Aug 20 02:54:25 2023
cpu : usr=4.85%, sys=0.00%, ctx=18, majf=0, minf=16
IO depths : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=66.7%, 8=33.3%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,8,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Seq-Read-1M-Q1-T1: (groupid=2, jobs=1): err= 0: pid=160248: Sun Aug 20 02:54:25 2023
read: IOPS=282, BW=282MiB/s (296MB/s)(1024MiB/3626msec)
slat (usec): min=316, max=404103, avg=3474.72, stdev=19094.04
clat (nsec): min=1948, max=420109, avg=5823.21, stdev=18705.45
lat (usec): min=318, max=404111, avg=3480.54, stdev=19094.42
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4],
| 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 7], 95.00th=[ 9],
| 99.00th=[ 73], 99.50th=[ 93], 99.90th=[ 302], 99.95th=[ 420],
| 99.99th=[ 420]
bw ( KiB/s): min=46369, max=376832, per=93.02%, avg=268988.43, stdev=141001.68, samples=7
iops : min= 45, max= 368, avg=262.43, stdev=137.69, samples=7
lat (usec) : 2=0.10%, 4=73.93%, 10=22.46%, 20=0.88%, 50=1.07%
lat (usec) : 100=1.07%, 250=0.29%, 500=0.20%
cpu : usr=0.47%, sys=1.30%, ctx=1030, majf=0, minf=266
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Seq-Write-1M-Q1-T1: (groupid=3, jobs=1): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160253: Sun Aug 20 02:54:25 2023
cpu : usr=2.08%, sys=0.00%, ctx=3, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Rand-Read-4K-Q32-T16: (groupid=4, jobs=16): err= 0: pid=160255: Sun Aug 20 02:54:25 2023
read: IOPS=93, BW=373KiB/s (382kB/s)(21.9MiB/60052msec)
slat (msec): min=51, max=482, avg=170.26, stdev=38.02
clat (usec): min=7, max=6693.0k, avg=5060592.48, stdev=983003.30
lat (msec): min=56, max=6868, avg=5230.85, stdev=985.62
clat percentiles (msec):
| 1.00th=[ 493], 5.00th=[ 3138], 10.00th=[ 4597], 20.00th=[ 4866],
| 30.00th=[ 5000], 40.00th=[ 5067], 50.00th=[ 5201], 60.00th=[ 5336],
| 70.00th=[ 5470], 80.00th=[ 5604], 90.00th=[ 5805], 95.00th=[ 6141],
| 99.00th=[ 6477], 99.50th=[ 6544], 99.90th=[ 6611], 99.95th=[ 6678],
| 99.99th=[ 6678]
bw ( KiB/s): min= 135, max= 520, per=98.60%, avg=368.58, stdev= 4.79, samples=1753
iops : min= 33, max= 130, avg=91.44, stdev= 1.22, samples=1753
lat (usec) : 10=0.14%, 20=0.14%
lat (msec) : 100=0.14%, 250=0.18%, 500=0.46%, 750=0.43%, 1000=0.37%
lat (msec) : 2000=1.62%, >=2000=96.50%
cpu : usr=0.01%, sys=0.02%, ctx=5720, majf=1, minf=674
IO depths : 1=0.3%, 2=0.6%, 4=1.1%, 8=2.3%, 16=4.6%, 32=91.1%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
issued rwts: total=5603,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Rand-Write-4K-Q32-T16: (groupid=5, jobs=16): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160333: Sun Aug 20 02:54:25 2023
cpu : usr=0.14%, sys=0.04%, ctx=1030, majf=0, minf=228
IO depths : 1=3.1%, 2=6.2%, 4=12.5%, 8=25.0%, 16=50.0%, 32=3.1%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=66.7%, 8=0.0%, 16=0.0%, 32=33.3%, 64=0.0%, >=64=0.0%
issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Rand-Read-4K-Q1-T1: (groupid=6, jobs=1): err= 0: pid=160350: Sun Aug 20 02:54:25 2023
read: IOPS=12, BW=50.1KiB/s (51.3kB/s)(3008KiB/60027msec)
slat (msec): min=34, max=132, avg=79.72, stdev=16.86
clat (usec): min=6, max=275, avg=10.55, stdev=18.51
lat (msec): min=34, max=132, avg=79.73, stdev=16.86
clat percentiles (usec):
| 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 8], 60.00th=[ 9],
| 70.00th=[ 9], 80.00th=[ 9], 90.00th=[ 9], 95.00th=[ 10],
| 99.00th=[ 118], 99.50th=[ 174], 99.90th=[ 277], 99.95th=[ 277],
| 99.99th=[ 277]
bw ( KiB/s): min= 39, max= 64, per=99.78%, avg=50.03, stdev= 5.85, samples=119
iops : min= 9, max= 16, avg=12.47, stdev= 1.48, samples=119
lat (usec) : 10=96.54%, 20=0.40%, 50=1.46%, 100=0.40%, 250=1.06%
lat (usec) : 500=0.13%
cpu : usr=0.03%, sys=0.03%, ctx=759, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=752,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Rand-Write-4K-Q1-T1: (groupid=7, jobs=1): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160467: Sun Aug 20 02:54:25 2023
cpu : usr=0.00%, sys=0.00%, ctx=3, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=248MiB/s (260MB/s), 248MiB/s-248MiB/s (260MB/s-260MB/s), io=1024MiB (1074MB), run=4135-4135msec
Run status group 1 (all jobs):
Run status group 2 (all jobs):
READ: bw=282MiB/s (296MB/s), 282MiB/s-282MiB/s (296MB/s-296MB/s), io=1024MiB (1074MB), run=3626-3626msec
Run status group 3 (all jobs):
Run status group 4 (all jobs):
READ: bw=373KiB/s (382kB/s), 373KiB/s-373KiB/s (382kB/s-382kB/s), io=21.9MiB (22.9MB), run=60052-60052msec
Run status group 5 (all jobs):
Run status group 6 (all jobs):
READ: bw=50.1KiB/s (51.3kB/s), 50.1KiB/s-50.1KiB/s (51.3kB/s-51.3kB/s), io=3008KiB (3080kB), run=60027-60027msec
Run status group 7 (all jobs):
また、fioだとS3のWrite性能が分からなかったので、ddコマンドでWrite性能を確認してみました。
以下コマンドをそれぞれのマウントポイントごとに10回実施した平均が表の結果となります。
ddコマンドによるWrite性能確認
dd if=/dev/zero of=/[EBS、EFS、S3マウントポイント]/.dd-benchmark bs=1M count=1000
| EBS dd結果 | EFS dd結果 | S3 dd結果 |
|---|---|---|
| 415MB/s | 316MB/s | 303MB/s |
ログアウトする際にコンソールが固まる
2023年8月現在、私がテストしたEC2環境ではSystems ManagerのSession ManagerでEC2に接続してS3をマウント→exitでEC2から抜けようとするとコンソールが固まってしまいました。
他のウィンドウから再度EC2にログインしてumountでマウント解除すると固まっていたコンソールも閉じられたので、Mountpoint for Amazon S3が悪さをしているようです。
マウントを実施したコンソール以外のウィンドウは特にexitで抜けても固まることはありませんでした。
また、Session Managerではなく、SSHで接続してS3をマウント→exitであればコンソールが固まることはなかったので、Session Manager＋Mountpoint for Amazon S3の組み合わせが問題を引き起こすようです。
もしSession Managerで操作を行う場合は上記の問題が発生するかと思われるので気をつけましょう。
おわりに
Mountpoint for Amazon S3はそもそもS3なので色々と制約はありますが、シーケンシャルReadに関してはEFSより良い性能が出たのは予想外でした。
また、書き込みについても300MB/s程度出たので、Mountpoint for Amazon S3の特性を理解して使うのであれば、EFSから置き換えることもできるのではないかと思います。
通常のファイルシステムとは勝手が違うので、導入は慎重に行う必要がありますが、うまく要件が合致するようであればS3の利点を活かしつつ、高いパフォーマンスも出すことができる保存領域としてシステムに導入できそうだと感じました。
今回紹介した内容以外の詳細はMountpoint for Amazon S3のGitHubのdocに色々と書かれているので、詳しい情報を知りたい場合は、公式のGitHubを確認してみてください。