Mountpoint for Amazon S3を使ってS3をマウントしてみる。

はじめに

以前から話題になっていたMountpoint for Amazon S3が一般公開となったそうなので、使い勝手を確認してみようと思います。

• Mountpoint for Amazon S3 の一般提供を開始
• Mountpoint for Amazon S3 – 一般公開されており、本番環境のワークロードにも対応可能
• GitHub - awslabs/mountpoint-s3

Mountpoint for Amazon S3とは

AWSのS3領域をEC2やECSにマウントして共有ディスクのように使えるアプリケーションです。

S3をマウントして使用できるアプリケーションは今までもs3fsやgoofysなどいくつかありましたが、Mountpoint for Amazon S3はAWSが公式で公開しているのが特徴です。

また、Mountpoint for Amazon S3はファイルシステムをマウントしたような感覚で使用することは出来ますが、POSIX規格には準拠していないため、POSIX規格に準拠したファイル操作と何が異なってくるのかを予め抑えておく必要があります。

Mountpoint for Amazon S3の使用

Mountpoint for Amazon S3は簡単な操作で使用することが出来ます。

以下に実際にS3をマウントして使用できるようにするまでの手順をまとめます。

• 事前設定
• Mountpoint for Amazon S3のインストール
• Mountpoint for Amazon S3のマウント
• Mountpoint for Amazon S3のマウント解除

事前設定

Mountpoint for Amazon S3は、要はクライアントアプリケーションを使ってEC2などのクライアントからS3に対してアクセスするものとなるので、当然クライアントからS3にアクセスするためのIAMユーザ・ロールの権限が必要となります。

以下は該当のS3バケットに対してフルアクセスするためのIAM権限の例です。

IAM権限の例

{
   "Version": "2012-10-17",
   "Statement": [
        {
            "Sid": "MountpointFullBucketAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::DOC-EXAMPLE-BUCKET"
            ]
        },
        {
            "Sid": "MountpointFullObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
            ]
        }
   ]
}

Mountpoint for Amazon S3のインストール

今回は適当なEC2を用意してS3をマウントしたいと思います。

Mountpoint for Amazon S3自体のインストールは簡単で、RPMパッケージをダウンロードしてyumでインストールするだけです。

公式GitHubにはOSごとに２種類のインストール方法が紹介されております。

Gravitonインスタンスにインストールする場合は、x86_64の部分をarm64に変えると良いそうです。

Mountpoint for Amazon S3のインストール（x86_64版Amazon Linux）

wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
sudo yum install -y ./mount-s3.rpm

Mountpoint for Amazon S3のインストール（x86_64版Ubuntu）

wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
sudo apt-get install -y ./mount-s3.deb

Mountpoint for Amazon S3のマウント

Mountpoint for Amazon S3をインストールすることで、mount-s3 [S3バケット名] [マウントポイント]の書式でS3をマウントできるようになります。

以下はmountpoint-for-s3-testバケットを、新たに作成したs3mountというマウントポイントにマウントする場合の例。

mount-s3コマンドによるS3マウント

mkdir s3mount
mount-s3 mountpoint-for-s3-test s3mount

以下のようにメッセージが表示されれば正常にマウントできています。

マウント時のメッセージ例

bucket mountpoint-for-s3-test is mounted at s3mount

また、上記のデフォルトマウントコマンドではS3上のオブジェクトを削除できないように制限されているため、削除も行いたい場合は--allow-deleteオプションを付与して以下のようにマウントします。

削除オプションを付与したS3マウント

mount-s3 --allow-delete mountpoint-for-s3-test s3mount

--prefix [プレフィックス名]/を指定することでS3バケット全体ではなく、特定のプレフィックス（フォルダ）部分のみマウントすることもできるので、特定の領域のみユーザに使わせるといったことも可能です。

なお、--prefixオプションを付与する際には最後に「/」を付与するようにしてください。

prefixオプションを付与したS3マウント

mount-s3 --allow-delete --prefix testdir/ mountpoint-for-s3-test s3mount

Mountpoint for Amazon S3のマウント解除

マウント解除は通常のumountで解除します。

S3マウントの解除

umount /s3mount

Mountpoint for Amazon S3の確認

上記手順でS3バケットをマウントすることが出来ましたが、一般的なファイルシステムをマウントした場合と比べてどのような違いがあるのかを、実際にコマンド実行しながら確認していきたいと思います。

• マウント時のコマンド確認
• ファイルリネーム
• ファイル削除
• ファイル追記
• ディレクトリの作成
• リンク作成

マウント時のコマンド確認

共有ディスクをマウントした場合等、dfコマンドを実行するとマウント済みの領域が表示されますが、Mountpoint for Amazon S3でマウントした場合はdfコマンドでは確認できません。

dfコマンドでの確認例

# df -k
Filesystem            1K-blocks    Used        Available Use% Mounted on
devtmpfs                   4096       0             4096   0% /dev
tmpfs                    486288       0           486288   0% /dev/shm
tmpfs                    194516    2904           191612   2% /run
/dev/xvda1              8310764 1640856          6669908  20% /
tmpfs                    486288       0           486288   0% /tmp
tmpfs                     97256       0            97256   0% /run/user/0
127.0.0.1:/    9007199254739968       0 9007199254739968   0% /efs
/dev/xvdf             104792064  763656        104028408   1% /io2

mountコマンドで確認することはできるので、マウントできているかを確認する場合はmountコマンドで確認するのが良いかと思います。

mountコマンドで確認した場合の例

# mount
（略）
mountpoint-s3 on /s3mount type fuse (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions)

ファイルリネーム

ファイルリネームには対応していないため実行すると以下のように表示されます。

ファイルリネーム例

# mv testfile testfile.bak
mv: cannot move 'testfile' to 'testfile.bak': Function not implemented

ファイル削除

前述の通り、--allow-deleteを付与しないと以下のように失敗します。

ファイルの削除例

# rm testfile
rm: remove regular empty file 'testfile'? y
rm: cannot remove 'testfile': Operation not permitted

また、--allow-deleteを付与しても、そもそもIAMロールやS3バケットポリシーで削除が許可されていないとできないので注意してください。

ファイル追記

ファイルの追記は以下のように失敗してしまいます。

ファイルの追記例

# echo test >> testfile
-bash: testfile: Operation not permitted

これは通常のS3操作を思い浮かべてもらえればイメージできるかと思いますが、そもそもS3に代表されるオブジェクトストレージは追記を想定したストレージではないのでMountpoint for Amazon S3でも追記することは出来ません。

一応viでの作成・編集は新規ファイルをコピーするような動作となるため出来なくはないようですが、編集する際に以下のようなメッセージが表示され、編集後、スワップファイルが残ってしまうので編集は出来ないものと考えておきましょう。

viでファイル編集した際のメッセージ

E325: ATTENTION
Found a swap file by the name ".testfile.swp"
          owned by: root   dated: Sat Aug 19 00:19:13 2023
         [cannot be read]
While opening file "testfile"
             dated: Sat Aug 19 00:19:27 2023
      NEWER than swap file!

(1) Another program may be editing the same file.  If this is the case,
    be careful not to end up with two different instances of the same
    file when making changes.  Quit, or continue with caution.
(2) An edit session for this file crashed.
    If this is the case, use ":recover" or "vim -r testfile"
    to recover the changes (see ":help recovery").
    If you did this already, delete the swap file ".testfile.swp"
    to avoid this message.

Swap file ".testfile.swp" already exists!
[O]pen Read-Only, (E)dit anyway, (R)ecover, (D)elete it, (Q)uit, (A)bort:

ディレクトリの作成

ディレクトリの作成は通常のディレクトリ作成と同様mkdirコマンドで作成することができます。

但し、ディレクトリの中身に何もファイルがない状態の場合、ローカル上での操作となり、S3にアップロードされません。

作成したディレクトリにファイルが格納されることでS3側にアップロードされる動作となるのでご注意ください。

S3領域にディレクトリのみ作成した場合のS3バケットの状態

# mkdir -p /s3mount/testdir1/testdir2
# tree /s3mount
/s3mount
└── testdir1
    └── testdir2

2 directories, 0 files
# aws s3 ls s3://mountpoint-for-s3-test/ --recursive
#

作成したディレクトリ配下にファイルを作成した場合のS3バケットの状態

# touch /s3mount/testdir1/testdir2/testfile
# tree /s3mount
/s3mount
└── testdir1
    └── testdir2
        └── testfile

2 directories, 1 file
# aws s3 ls s3://mountpoint-for-s3-test/ --recursive
2023-08-19 01:47:52          0 testdir/testdir1/testdir2/testfile

リンク作成

lnコマンドで作成できるハードリンク、シンボリックリンクはどちらもサポートされておりません。

シンボリックリンクの作成例

# ln -s /var/log testlink
ln: failed to create symbolic link 'testlink': Operation not permitted

Mountpoint for Amazon S3のベンチマーク

Mountpoint for Amazon S3でマウントした領域はどの程度の性能が出るのか、ベンチマークをしてみようと思います。

ベンチマーク準備

今回はEBS、EFS、Mountpoint for Amazon S3の３つを比較してみようと思います。

以下それぞれのタイプ等。

ベンチマーク対象	設定・タイプ	マウントポイント
EBS	ボリュームタイプ：プロビジョンドIOPS（io2） サイズ：100GB IOPS：3000	/io2
EFS	パフォーマンスモード：汎用 スループットモード：伸縮自在 EFSマウントヘルパー使用	/efs
Mountpoint for Amazon S3	ストレージクラス：スタンダード --allow-deleteオプション付与	/s3mount

また、ベンチマークにはAmazon Linux 2023のt3.smallを使用します。

fioの設定

ちょうど「EBSボリュームのベンチマーク」のドキュメントにfioを使用したベンチマーク方法が書かれていたので、こちらを参考にCrystalDiskMark風なパターンでベンチマークしてみようと思います。

ベンチマークで使用したfioの設定は以下となります。

fio.config

[global]
ioengine=libaio
iodepth=1
size=1g
direct=1
runtime=60
directory=/s3mount
filename=.fio-benchmark
norandommap
group_reporting
stonewall

[Seq-Read-1M-Q8-T1]
rw=read
bs=1m
iodepth=8
numjobs=1

[Seq-Write-1M-Q8-T1]
rw=write
bs=1m
iodepth=8
numjobs=1

[Seq-Read-1M-Q1-T1]
rw=read
bs=1m
iodepth=1
numjobs=1

[Seq-Write-1M-Q1-T1]
rw=write
bs=1m
iodepth=1
numjobs=1

[Rand-Read-4K-Q32-T16]
rw=randread
bs=4k
iodepth=32
numjobs=16

[Rand-Write-4K-Q32-T16]
rw=randwrite
bs=4k
iodepth=32
numjobs=16

[Rand-Read-4K-Q1-T1]
rw=randread
bs=4k
iodepth=1
numjobs=1

[Rand-Write-4K-Q1-T1]
rw=randwrite
bs=4k
iodepth=1
numjobs=1

directory部分をEBSやEFS、Mountpoint for Amazon S3をマウントしたパスに書き換えて以下のように実行します。

fioの実行

fio fio.config

ベンチマーク実行結果

ベンチマークを行った結果が以下となります。

Mountpoint for Amazon S3のWriteの性能を「ー」としておりますが、上述の通り追記等の処理が行えないためfioを実行してもエラーとなるためです。

EBSのランダムRead・WriteはIOPSの上限に引っかかっているかと思うので、実際はもっと性能が出るかと思いますが、今回はMountpoint for Amazon S3の評価が本題なのでこのままとします。

パターン	EBS転送速度	EBS IOPS	EFS転送速度	EFS IOPS	S3転送速度	S3 IOPS
シーケンシャルRead 1M Q8 T1	344MB/s	327	213MB/s	203	260MB/s	247
シーケンシャルWrite 1M Q8 T1	280MB/s	267	144MB/s	137	ー	ー
シーケンシャルRead 1M Q1 T1	286MB/s	272	47.2MB/s	45	296MB/s	282
シーケンシャルWrite 1M Q1 T1	285MB/s	272	34.9MB/s	33	ー	ー
ランダムRead 4K Q32 T16	12.5MB/s	3049	43.7MB/s	10.7k	382kB/s	93
ランダムWrite 4K Q32 T16	12.3MB/s	3015	19.3MB/s	4702	ー	ー
ランダムRead 4K Q1 T1	12.4MB/s	3019	1241kB/s	302	51.3kB/s	12
ランダムWrite 4K Q1 T1	12.3MB/s	3014	442kB/s	107	ー	ー

各パターンの詳細を見たい方は以下展開して確認してください。

EBS結果（展開してください）

EBSのfio結果

Seq-Read-1M-Q8-T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Write-1M-Q8-T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Read-1M-Q1-T1: (g=2): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seq-Write-1M-Q1-T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Rand-Read-4K-Q32-T16: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Write-4K-Q32-T16: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Read-4K-Q1-T1: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Rand-Write-4K-Q1-T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32
Starting 38 processes
Jobs: 1 (f=0): [_(37),f(1)][100.0%][w=11.0MiB/s][w=2804 IOPS][eta 00m:00s]
Seq-Read-1M-Q8-T1: (groupid=0, jobs=1): err= 0: pid=156247: Sun Aug 20 02:14:52 2023
  read: IOPS=327, BW=328MiB/s (344MB/s)(1024MiB/3122msec)
    slat (usec): min=23, max=6936, avg=2013.71, stdev=1536.82
    clat (usec): min=2149, max=53106, avg=22330.75, stdev=11665.09
     lat (usec): min=2180, max=56222, avg=24344.46, stdev=12859.52
    clat percentiles (usec):
     |  1.00th=[ 2573],  5.00th=[ 3032], 10.00th=[ 3458], 20.00th=[ 3949],
     | 30.00th=[28443], 40.00th=[28705], 50.00th=[28967], 60.00th=[28967],
     | 70.00th=[29230], 80.00th=[29230], 90.00th=[30016], 95.00th=[32113],
     | 99.00th=[33162], 99.50th=[36963], 99.90th=[49021], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=253952, max=647739, per=99.21%, avg=333220.60, stdev=175823.13, samples=5
   iops        : min=  248, max=  632, avg=325.20, stdev=171.51, samples=5
  lat (msec)   : 4=20.80%, 10=6.74%, 20=0.29%, 50=72.07%, 100=0.10%
  cpu          : usr=0.54%, sys=1.86%, ctx=1651, majf=0, minf=2059
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Write-1M-Q8-T1: (groupid=1, jobs=1): err= 0: pid=156251: Sun Aug 20 02:14:52 2023
  write: IOPS=267, BW=267MiB/s (280MB/s)(1024MiB/3830msec); 0 zone resets
    slat (usec): min=44, max=16322, avg=2231.71, stdev=1718.08
    clat (usec): min=1655, max=53202, avg=27652.55, stdev=8100.48
     lat (usec): min=1732, max=56471, avg=29884.27, stdev=8665.53
    clat percentiles (usec):
     |  1.00th=[ 3163],  5.00th=[ 3458], 10.00th=[21890], 20.00th=[28705],
     | 30.00th=[28967], 40.00th=[28967], 50.00th=[28967], 60.00th=[28967],
     | 70.00th=[28967], 80.00th=[31589], 90.00th=[32113], 95.00th=[34866],
     | 99.00th=[46400], 99.50th=[48497], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=253952, max=354837, per=98.29%, avg=269094.43, stdev=37818.95, samples=7
   iops        : min=  248, max=  346, avg=262.57, stdev=36.80, samples=7
  lat (msec)   : 2=0.10%, 4=7.81%, 10=0.39%, 20=1.46%, 50=89.94%
  lat (msec)   : 100=0.29%
  cpu          : usr=1.88%, sys=1.41%, ctx=1678, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Read-1M-Q1-T1: (groupid=2, jobs=1): err= 0: pid=156257: Sun Aug 20 02:14:52 2023
  read: IOPS=272, BW=273MiB/s (286MB/s)(1024MiB/3751msec)
    slat (usec): min=20, max=4781, avg=47.30, stdev=152.79
    clat (usec): min=684, max=7294, avg=3609.84, stdev=1057.06
     lat (usec): min=746, max=7328, avg=3657.14, stdev=1058.08
    clat percentiles (usec):
     |  1.00th=[  750],  5.00th=[  791], 10.00th=[  865], 20.00th=[ 3916],
     | 30.00th=[ 3949], 40.00th=[ 3982], 50.00th=[ 3982], 60.00th=[ 3982],
     | 70.00th=[ 3982], 80.00th=[ 4015], 90.00th=[ 4015], 95.00th=[ 4047],
     | 99.00th=[ 5014], 99.50th=[ 5669], 99.90th=[ 7242], 99.95th=[ 7308],
     | 99.99th=[ 7308]
   bw (  KiB/s): min=253952, max=428740, per=100.00%, avg=279652.00, stdev=65747.39, samples=7
   iops        : min=  248, max=  418, avg=272.86, stdev=64.01, samples=7
  lat (usec)   : 750=0.68%, 1000=9.96%
  lat (msec)   : 2=1.27%, 4=68.75%, 10=19.34%
  cpu          : usr=0.88%, sys=0.91%, ctx=1025, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seq-Write-1M-Q1-T1: (groupid=3, jobs=1): err= 0: pid=156262: Sun Aug 20 02:14:52 2023
  write: IOPS=272, BW=272MiB/s (285MB/s)(1024MiB/3764msec); 0 zone resets
    slat (usec): min=42, max=1731, avg=105.84, stdev=76.87
    clat (usec): min=1130, max=8655, avg=3564.31, stdev=965.57
     lat (usec): min=1172, max=8732, avg=3670.15, stdev=972.52
    clat percentiles (usec):
     |  1.00th=[ 1156],  5.00th=[ 1205], 10.00th=[ 1270], 20.00th=[ 3818],
     | 30.00th=[ 3884], 40.00th=[ 3884], 50.00th=[ 3916], 60.00th=[ 3916],
     | 70.00th=[ 3949], 80.00th=[ 3949], 90.00th=[ 3982], 95.00th=[ 4015],
     | 99.00th=[ 4883], 99.50th=[ 5800], 99.90th=[ 6849], 99.95th=[ 8717],
     | 99.99th=[ 8717]
   bw (  KiB/s): min=253952, max=387786, per=98.28%, avg=273801.43, stdev=50270.01, samples=7
   iops        : min=  248, max=  378, avg=267.14, stdev=48.89, samples=7
  lat (msec)   : 2=13.38%, 4=80.18%, 10=6.45%
  cpu          : usr=2.02%, sys=0.93%, ctx=1025, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Read-4K-Q32-T16: (groupid=4, jobs=16): err= 0: pid=156267: Sun Aug 20 02:14:52 2023
  read: IOPS=3049, BW=11.9MiB/s (12.5MB/s)(715MiB/60027msec)
    slat (nsec): min=1924, max=128039k, avg=5238465.70, stdev=9531041.06
    clat (usec): min=181, max=474989, avg=162614.04, stdev=51900.61
     lat (usec): min=189, max=475000, avg=167852.51, stdev=52879.82
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[   80], 10.00th=[  104], 20.00th=[  126],
     | 30.00th=[  140], 40.00th=[  153], 50.00th=[  163], 60.00th=[  176],
     | 70.00th=[  188], 80.00th=[  203], 90.00th=[  226], 95.00th=[  247],
     | 99.00th=[  284], 99.50th=[  300], 99.90th=[  342], 99.95th=[  363],
     | 99.99th=[  426]
   bw (  KiB/s): min= 7305, max=31279, per=99.67%, avg=12159.87, stdev=168.93, samples=1904
   iops        : min= 1826, max= 7814, avg=3037.86, stdev=42.21, samples=1904
  lat (usec)   : 250=0.04%, 500=0.10%, 750=0.03%, 1000=0.03%
  lat (msec)   : 2=0.20%, 4=0.62%, 10=0.45%, 20=0.27%, 50=0.92%
  lat (msec)   : 100=6.51%, 250=86.52%, 500=4.31%
  cpu          : usr=0.12%, sys=0.20%, ctx=135988, majf=0, minf=677
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=183078,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Write-4K-Q32-T16: (groupid=5, jobs=16): err= 0: pid=156404: Sun Aug 20 02:14:52 2023
  write: IOPS=3015, BW=11.8MiB/s (12.3MB/s)(707MiB/60027msec); 0 zone resets
    slat (usec): min=2, max=296642, avg=5298.40, stdev=15383.26
    clat (usec): min=138, max=590686, avg=164448.26, stdev=71623.19
     lat (usec): min=242, max=590689, avg=169746.66, stdev=72696.41
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   73], 10.00th=[   86], 20.00th=[  104],
     | 30.00th=[  120], 40.00th=[  136], 50.00th=[  153], 60.00th=[  171],
     | 70.00th=[  194], 80.00th=[  220], 90.00th=[  259], 95.00th=[  296],
     | 99.00th=[  380], 99.50th=[  409], 99.90th=[  481], 99.95th=[  493],
     | 99.99th=[  542]
   bw (  KiB/s): min= 5264, max=20646, per=99.82%, avg=12038.22, stdev=172.27, samples=1897
   iops        : min= 1316, max= 5156, avg=3007.51, stdev=43.04, samples=1897
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.05%, 1000=0.09%
  lat (msec)   : 2=0.07%, 4=0.02%, 10=0.08%, 20=0.05%, 50=1.03%
  lat (msec)   : 100=16.45%, 250=70.06%, 500=12.04%, 750=0.03%
  cpu          : usr=0.10%, sys=0.18%, ctx=162798, majf=0, minf=159
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,180986,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Read-4K-Q1-T1: (groupid=6, jobs=1): err= 0: pid=156539: Sun Aug 20 02:14:52 2023
  read: IOPS=3019, BW=11.8MiB/s (12.4MB/s)(708MiB/60001msec)
    slat (usec): min=2, max=12040, avg= 6.52, stdev=46.70
    clat (nsec): min=1999, max=15429k, avg=322639.92, stdev=216497.01
     lat (usec): min=89, max=15432, avg=329.16, stdev=220.85
    clat percentiles (usec):
     |  1.00th=[  102],  5.00th=[  113], 10.00th=[  123], 20.00th=[  161],
     | 30.00th=[  297], 40.00th=[  314], 50.00th=[  322], 60.00th=[  330],
     | 70.00th=[  338], 80.00th=[  355], 90.00th=[  506], 95.00th=[  537],
     | 99.00th=[  889], 99.50th=[ 1221], 99.90th=[ 2835], 99.95th=[ 3949],
     | 99.99th=[ 6325]
   bw (  KiB/s): min=11823, max=19900, per=99.97%, avg=12075.57, stdev=727.08, samples=118
   iops        : min= 2955, max= 4975, avg=3018.85, stdev=181.78, samples=118
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.64%, 250=23.47%, 500=64.79%, 750=9.51%, 1000=0.79%
  lat (msec)   : 2=0.58%, 4=0.16%, 10=0.04%, 20=0.01%
  cpu          : usr=1.45%, sys=2.80%, ctx=181235, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=181188,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Write-4K-Q1-T1: (groupid=7, jobs=1): err= 0: pid=156603: Sun Aug 20 02:14:52 2023
  write: IOPS=3014, BW=11.8MiB/s (12.3MB/s)(707MiB/60001msec); 0 zone resets
    slat (usec): min=2, max=7151, avg= 7.62, stdev=40.18
    clat (usec): min=2, max=9981, avg=321.91, stdev=193.70
     lat (usec): min=177, max=9987, avg=329.54, stdev=197.56
    clat percentiles (usec):
     |  1.00th=[  190],  5.00th=[  200], 10.00th=[  210], 20.00th=[  258],
     | 30.00th=[  302], 40.00th=[  314], 50.00th=[  322], 60.00th=[  326],
     | 70.00th=[  334], 80.00th=[  343], 90.00th=[  355], 95.00th=[  392],
     | 99.00th=[  766], 99.50th=[ 1254], 99.90th=[ 3359], 99.95th=[ 4113],
     | 99.99th=[ 5997]
   bw (  KiB/s): min=11512, max=17744, per=100.00%, avg=12071.06, stdev=547.70, samples=119
   iops        : min= 2878, max= 4436, avg=3017.76, stdev=136.93, samples=119
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=19.19%, 500=78.69%, 750=1.07%, 1000=0.35%
  lat (msec)   : 2=0.42%, 4=0.20%, 10=0.06%
  cpu          : usr=1.38%, sys=2.23%, ctx=181109, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,180898,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=1024MiB (1074MB), run=3122-3122msec

Run status group 1 (all jobs):
  WRITE: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=1024MiB (1074MB), run=3830-3830msec

Run status group 2 (all jobs):
   READ: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=1024MiB (1074MB), run=3751-3751msec

Run status group 3 (all jobs):
  WRITE: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=1024MiB (1074MB), run=3764-3764msec

Run status group 4 (all jobs):
   READ: bw=11.9MiB/s (12.5MB/s), 11.9MiB/s-11.9MiB/s (12.5MB/s-12.5MB/s), io=715MiB (750MB), run=60027-60027msec

Run status group 5 (all jobs):
  WRITE: bw=11.8MiB/s (12.3MB/s), 11.8MiB/s-11.8MiB/s (12.3MB/s-12.3MB/s), io=707MiB (741MB), run=60027-60027msec

Run status group 6 (all jobs):
   READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=708MiB (742MB), run=60001-60001msec

Run status group 7 (all jobs):
  WRITE: bw=11.8MiB/s (12.3MB/s), 11.8MiB/s-11.8MiB/s (12.3MB/s-12.3MB/s), io=707MiB (741MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=372458/369537, merge=0/0, ticks=3817558/3740979, in_queue=7558537, util=99.15%

EFS結果（展開してください）

EFSのfio結果

Seq-Read-1M-Q8-T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Write-1M-Q8-T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Read-1M-Q1-T1: (g=2): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seq-Write-1M-Q1-T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Rand-Read-4K-Q32-T16: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Write-4K-Q32-T16: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Read-4K-Q1-T1: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Rand-Write-4K-Q1-T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32
Starting 38 processes
Seq-Read-1M-Q8-T1: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [_(37),w(1)][13.1%][w=440KiB/s][w=110 IOPS][eta 34m:04s]
Seq-Read-1M-Q8-T1: (groupid=0, jobs=1): err= 0: pid=165964: Sun Aug 20 03:56:48 2023
  read: IOPS=203, BW=203MiB/s (213MB/s)(1024MiB/5037msec)
    slat (usec): min=49, max=2487, avg=139.14, stdev=149.19
    clat (msec): min=16, max=236, avg=39.04, stdev=24.60
     lat (msec): min=16, max=236, avg=39.18, stdev=24.60
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[   22], 10.00th=[   23], 20.00th=[   26],
     | 30.00th=[   29], 40.00th=[   31], 50.00th=[   33], 60.00th=[   35],
     | 70.00th=[   39], 80.00th=[   44], 90.00th=[   56], 95.00th=[   91],
     | 99.00th=[  148], 99.50th=[  163], 99.90th=[  228], 99.95th=[  236],
     | 99.99th=[  236]
   bw (  KiB/s): min=175028, max=235991, per=100.00%, avg=209907.00, stdev=18602.14, samples=9
   iops        : min=  170, max=  230, avg=204.67, stdev=18.25, samples=9
  lat (msec)   : 20=2.64%, 50=84.86%, 100=8.11%, 250=4.39%
  cpu          : usr=0.10%, sys=1.89%, ctx=1552, majf=0, minf=2058
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Write-1M-Q8-T1: (groupid=1, jobs=1): err= 0: pid=165970: Sun Aug 20 03:56:48 2023
  write: IOPS=137, BW=137MiB/s (144MB/s)(1024MiB/7450msec); 0 zone resets
    slat (usec): min=91, max=3787, avg=186.43, stdev=202.13
    clat (msec): min=20, max=302, avg=57.91, stdev=29.14
     lat (msec): min=21, max=302, avg=58.09, stdev=29.14
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   29], 10.00th=[   32], 20.00th=[   37],
     | 30.00th=[   45], 40.00th=[   51], 50.00th=[   56], 60.00th=[   61],
     | 70.00th=[   66], 80.00th=[   71], 90.00th=[   79], 95.00th=[   87],
     | 99.00th=[  186], 99.50th=[  241], 99.90th=[  288], 99.95th=[  305],
     | 99.99th=[  305]
   bw (  KiB/s): min=126470, max=151552, per=99.97%, avg=140701.07, stdev=8646.52, samples=14
   iops        : min=  123, max=  148, avg=137.29, stdev= 8.48, samples=14
  lat (msec)   : 50=39.36%, 100=57.42%, 250=2.73%, 500=0.49%
  cpu          : usr=0.78%, sys=1.53%, ctx=1046, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Read-1M-Q1-T1: (groupid=2, jobs=1): err= 0: pid=165979: Sun Aug 20 03:56:48 2023
  read: IOPS=45, BW=45.0MiB/s (47.2MB/s)(1024MiB/22740msec)
    slat (usec): min=61, max=7907, avg=100.47, stdev=300.69
    clat (usec): min=13306, max=97495, avg=22095.31, stdev=7013.49
     lat (usec): min=13782, max=97571, avg=22195.78, stdev=7025.72
    clat percentiles (usec):
     |  1.00th=[16581],  5.00th=[17433], 10.00th=[17957], 20.00th=[18482],
     | 30.00th=[19006], 40.00th=[19530], 50.00th=[20055], 60.00th=[20841],
     | 70.00th=[21890], 80.00th=[23987], 90.00th=[27657], 95.00th=[31589],
     | 99.00th=[53740], 99.50th=[61080], 99.90th=[95945], 99.95th=[96994],
     | 99.99th=[96994]
   bw (  KiB/s): min=28500, max=53354, per=99.86%, avg=46045.75, stdev=5430.42, samples=44
   iops        : min=   27, max=   52, avg=44.84, stdev= 5.40, samples=44
  lat (msec)   : 20=47.36%, 50=51.07%, 100=1.56%
  cpu          : usr=0.05%, sys=0.38%, ctx=1044, majf=0, minf=265
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seq-Write-1M-Q1-T1: (groupid=3, jobs=1): err= 0: pid=166006: Sun Aug 20 03:56:48 2023
  write: IOPS=33, BW=33.3MiB/s (34.9MB/s)(1024MiB/30761msec); 0 zone resets
    slat (usec): min=102, max=3276, avg=179.53, stdev=153.81
    clat (msec): min=20, max=422, avg=29.85, stdev=18.51
     lat (msec): min=21, max=423, avg=30.03, stdev=18.51
    clat percentiles (msec):
     |  1.00th=[   24],  5.00th=[   25], 10.00th=[   26], 20.00th=[   26],
     | 30.00th=[   27], 40.00th=[   28], 50.00th=[   28], 60.00th=[   29],
     | 70.00th=[   30], 80.00th=[   31], 90.00th=[   34], 95.00th=[   36],
     | 99.00th=[   48], 99.50th=[  113], 99.90th=[  359], 99.95th=[  422],
     | 99.99th=[  422]
   bw (  KiB/s): min= 6144, max=38989, per=99.77%, avg=34011.23, stdev=5932.22, samples=60
   iops        : min=    6, max=   38, avg=33.12, stdev= 5.81, samples=60
  lat (msec)   : 50=99.02%, 100=0.39%, 250=0.39%, 500=0.20%
  cpu          : usr=0.29%, sys=0.29%, ctx=1033, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Read-4K-Q32-T16: (groupid=4, jobs=16): err= 0: pid=166042: Sun Aug 20 03:56:48 2023
  read: IOPS=10.7k, BW=41.7MiB/s (43.7MB/s)(2504MiB/60037msec)
    slat (nsec): min=1881, max=9893.2k, avg=23118.50, stdev=73466.89
    clat (msec): min=4, max=178, avg=47.92, stdev= 9.80
     lat (msec): min=4, max=178, avg=47.94, stdev= 9.80
    clat percentiles (msec):
     |  1.00th=[   33],  5.00th=[   37], 10.00th=[   39], 20.00th=[   41],
     | 30.00th=[   43], 40.00th=[   45], 50.00th=[   47], 60.00th=[   48],
     | 70.00th=[   51], 80.00th=[   54], 90.00th=[   60], 95.00th=[   65],
     | 99.00th=[   79], 99.50th=[   87], 99.90th=[  146], 99.95th=[  157],
     | 99.99th=[  167]
   bw (  KiB/s): min=24408, max=55360, per=100.00%, avg=42738.87, stdev=325.55, samples=1904
   iops        : min= 6090, max=13840, avg=10682.66, stdev=81.36, samples=1904
  lat (msec)   : 10=0.01%, 20=0.02%, 50=67.94%, 100=31.83%, 250=0.21%
  cpu          : usr=0.38%, sys=0.94%, ctx=1035226, majf=0, minf=652
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=641041,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Write-4K-Q32-T16: (groupid=5, jobs=16): err= 0: pid=166176: Sun Aug 20 03:56:48 2023
  write: IOPS=4702, BW=18.4MiB/s (19.3MB/s)(1104MiB/60099msec); 0 zone resets
    slat (usec): min=2, max=8942, avg=32.17, stdev=100.44
    clat (msec): min=10, max=1307, avg=108.81, stdev=52.17
     lat (msec): min=10, max=1307, avg=108.84, stdev=52.17
    clat percentiles (msec):
     |  1.00th=[   94],  5.00th=[   96], 10.00th=[   97], 20.00th=[  100],
     | 30.00th=[  101], 40.00th=[  103], 50.00th=[  104], 60.00th=[  105],
     | 70.00th=[  107], 80.00th=[  110], 90.00th=[  115], 95.00th=[  121],
     | 99.00th=[  161], 99.50th=[  634], 99.90th=[  953], 99.95th=[  961],
     | 99.99th=[  961]
   bw (  KiB/s): min= 6752, max=20938, per=100.00%, avg=19142.83, stdev=125.47, samples=1856
   iops        : min= 1688, max= 5234, avg=4782.65, stdev=31.37, samples=1856
  lat (msec)   : 20=0.01%, 50=0.05%, 100=26.98%, 250=72.25%, 500=0.16%
  lat (msec)   : 750=0.22%, 1000=0.33%, 2000=0.01%
  cpu          : usr=0.20%, sys=0.39%, ctx=523481, majf=0, minf=151
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,282635,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Read-4K-Q1-T1: (groupid=6, jobs=1): err= 0: pid=166311: Sun Aug 20 03:56:48 2023
  read: IOPS=302, BW=1212KiB/s (1241kB/s)(71.0MiB/60002msec)
    slat (usec): min=8, max=3208, avg=17.99, stdev=56.87
    clat (usec): min=594, max=29700, avg=3274.09, stdev=1037.00
     lat (usec): min=606, max=29721, avg=3292.08, stdev=1040.45
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[ 2507], 10.00th=[ 2638], 20.00th=[ 2769],
     | 30.00th=[ 2868], 40.00th=[ 2966], 50.00th=[ 3064], 60.00th=[ 3261],
     | 70.00th=[ 3458], 80.00th=[ 3687], 90.00th=[ 4228], 95.00th=[ 5145],
     | 99.00th=[ 6915], 99.50th=[ 8094], 99.90th=[10683], 99.95th=[11731],
     | 99.99th=[16909]
   bw (  KiB/s): min= 1029, max= 1376, per=100.00%, avg=1212.30, stdev=63.28, samples=119
   iops        : min=  257, max=  344, avg=302.95, stdev=15.85, samples=119
  lat (usec)   : 750=2.94%, 1000=0.69%
  lat (msec)   : 2=0.16%, 4=83.86%, 10=12.19%, 20=0.16%, 50=0.01%
  cpu          : usr=0.35%, sys=0.53%, ctx=18381, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=18180,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Write-4K-Q1-T1: (groupid=7, jobs=1): err= 0: pid=166374: Sun Aug 20 03:56:48 2023
  write: IOPS=107, BW=432KiB/s (442kB/s)(25.3MiB/60008msec); 0 zone resets
    slat (usec): min=13, max=1871, avg=23.89, stdev=38.04
    clat (usec): min=6872, max=74459, avg=9229.58, stdev=1743.69
     lat (usec): min=6941, max=74477, avg=9253.46, stdev=1744.96
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[ 7635], 10.00th=[ 7832], 20.00th=[ 8094],
     | 30.00th=[ 8291], 40.00th=[ 8586], 50.00th=[ 8848], 60.00th=[ 9241],
     | 70.00th=[ 9634], 80.00th=[10028], 90.00th=[10945], 95.00th=[12387],
     | 99.00th=[14484], 99.50th=[15795], 99.90th=[20579], 99.95th=[24773],
     | 99.99th=[74974]
   bw (  KiB/s): min=  368, max=  456, per=100.00%, avg=432.16, stdev=13.20, samples=119
   iops        : min=   92, max=  114, avg=108.02, stdev= 3.30, samples=119
  lat (msec)   : 10=79.13%, 20=20.72%, 50=0.14%, 100=0.02%
  cpu          : usr=0.24%, sys=0.22%, ctx=6553, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,6478,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=203MiB/s (213MB/s), 203MiB/s-203MiB/s (213MB/s-213MB/s), io=1024MiB (1074MB), run=5037-5037msec

Run status group 1 (all jobs):
  WRITE: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=1024MiB (1074MB), run=7450-7450msec

Run status group 2 (all jobs):
   READ: bw=45.0MiB/s (47.2MB/s), 45.0MiB/s-45.0MiB/s (47.2MB/s-47.2MB/s), io=1024MiB (1074MB), run=22740-22740msec

Run status group 3 (all jobs):
  WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=1024MiB (1074MB), run=30761-30761msec

Run status group 4 (all jobs):
   READ: bw=41.7MiB/s (43.7MB/s), 41.7MiB/s-41.7MiB/s (43.7MB/s-43.7MB/s), io=2504MiB (2626MB), run=60037-60037msec

Run status group 5 (all jobs):
  WRITE: bw=18.4MiB/s (19.3MB/s), 18.4MiB/s-18.4MiB/s (19.3MB/s-19.3MB/s), io=1104MiB (1158MB), run=60099-60099msec

Run status group 6 (all jobs):
   READ: bw=1212KiB/s (1241kB/s), 1212KiB/s-1212KiB/s (1241kB/s-1241kB/s), io=71.0MiB (74.5MB), run=60002-60002msec

Run status group 7 (all jobs):
  WRITE: bw=432KiB/s (442kB/s), 432KiB/s-432KiB/s (442kB/s-442kB/s), io=25.3MiB (26.5MB), run=60008-60008msec

S3結果（展開してください）

S3のfio結果

Seq-Read-1M-Q8-T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Write-1M-Q8-T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Read-1M-Q1-T1: (g=2): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seq-Write-1M-Q1-T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Rand-Read-4K-Q32-T16: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Write-4K-Q32-T16: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Read-4K-Q1-T1: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Rand-Write-4K-Q1-T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32
Starting 38 processes
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=0, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1048576, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=2097152, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=3145728, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=4194304, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=5242880, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=6291456, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=7340032, buflen=1048576
fio: pid=160247, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=0, buflen=1048576
fio: pid=160253, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=935215104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=857464832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=468156416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=985276416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=8654848, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1052733440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=18911232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=119975936, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=176386048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=547172352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=81301504, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=459362304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=710455296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=295763968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1033744384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=117538816, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=495435776, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=669491200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=294543360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=384147456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=914653184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=477323264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1014996992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=792219648, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=912949248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=542818304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=746360832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=936366080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=251334656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=784572416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=70127616, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=518848512, buflen=4096
fio: pid=160333, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=263307264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=615055360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=532480000, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=384966656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=421335040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=61181952, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=708857856, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=121028608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=313151488, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=969424896, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=824680448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=780730368, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=976297984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=478814208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=816156672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=89661440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=14708736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=16232448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=305840128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=32333824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=642404352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=538820608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=858578944, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=930562048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=578523136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=753684480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1038864384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=864620544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=504500224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=402882560, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=838029312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=954761216, buflen=4096
fio: pid=160338, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=523722752, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=542588928, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=308756480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=268566528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=293478400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=146636800, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=371425280, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=955932672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=382050304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=310792192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=775139328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=548966400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=71241728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=941813760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=483074048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=798691328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=669417472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=886464512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12976128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=669818880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=694996992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=276897792, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=837181440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=102940672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=76701696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=840101888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=243916800, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=145715200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=674836480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=743108608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=661725184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=438169600, buflen=4096
fio: pid=160335, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=934752256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=99217408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=425132032, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=806121472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=990658560, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=346288128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=953012224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=28512256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=499052544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=478146560, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=823971840, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=776744960, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=754278400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=19656704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=429019136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=959897600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12582912, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=697966592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12124160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=813838336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=506232832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1027772416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=543240192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=220508160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=342061056, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=364220416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=304779264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=954494976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=445227008, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=58896384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=610508800, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=981893120, buflen=4096
fio: pid=160348, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=8601600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=878317568, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=252477440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=479391744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=311762944, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=968626176, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=132915200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=731619328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=813502464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=238047232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=303321088, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=870821888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=412164096, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=488108032, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=219828224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=409120768, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=645390336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=716910592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=960286720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=401412096, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=823185408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=435511296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1062674432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=922300416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=36704256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=166371328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1007411200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=102277120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=385798144, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=764690432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=281698304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1055588352, buflen=4096
fio: pid=160346, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=941694976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=658329600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=696217600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=885084160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=448086016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=755429376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=52625408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1004957696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=514899968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=583254016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=663240704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=23285760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=765550592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=912629760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=719757312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=533606400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=888299520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=831922176, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=581382144, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=699023360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1036546048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=37203968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=428335104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1067794432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=894111744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=179408896, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1041235968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=395902976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1019281408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=923275264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=131219456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=359477248, buflen=4096
fio: pid=160342, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=239673344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=440659968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=485097472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=549363712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=230699008, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=588984320, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=921407488, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=4083712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=535695360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=778104832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=181014528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=579887104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=976420864, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=194121728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1031229440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=739954688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=357748736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=407511040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=61419520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=941998080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=712503296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=996265984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=494002176, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=562319360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=600989696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=924049408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12562432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=700940288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=345415680, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=171368448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=916762624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=29659136, buflen=4096
fio: pid=160340, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=977674240, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=284614656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=840081408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=615800832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=946143232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=839675904, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=585740288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=488005632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=976441344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=573325312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1002610688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=660320256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=390197248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=315158528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=652472320, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=578666496, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=108425216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=151556096, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=785080320, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=466202624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=653971456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=129118208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=771239936, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=678895616, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=581734400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=824451072, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=615186432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030008832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=324722688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=232640512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=775360512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=440455168, buflen=4096
fio: pid=160339, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=468844544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=816947200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=331497472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030017024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=289689600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=223203328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=801005568, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=518500352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=688750592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=674770944, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=907816960, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1002545152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=44814336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=502349824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=216772608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=322326528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=959201280, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=774053888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=942166016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=149180416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=766337024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=159719424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=511873024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=492613632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=124293120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=967725056, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=829657088, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030090752, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=934739968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=275898368, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=248991744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=373313536, buflen=4096
fio: pid=160343, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=249245696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=618217472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=426070016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=714809344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=770711552, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=135536640, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=119123968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=611643392, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=596529152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=678940672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=642818048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=351408128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=625250304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=970063872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=596107264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=657723392, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=168054784, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=89559040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=149348352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=411435008, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=879968256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1021427712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=288948224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=204644352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=219381760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=814739456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=11239424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=979767296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=127823872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=99774464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=260157440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=109899776, buflen=4096
fio: pid=160334, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=628183040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=213262336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=919969792, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=970616832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=883093504, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1224704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=534097920, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=638631936, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=908935168, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=413306880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=901947392, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=478990336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=831348736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=488243200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1055608832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030279168, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1061765120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=853839872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=117583872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1066917888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=160763904, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=939462656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=317804544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=954675200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=695492608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=305479680, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=670142464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=855990272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=376033280, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=180854784, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=764383232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=213557248, buflen=4096
fio: pid=160344, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=171302912, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=426074112, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=631111680, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1049264128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=768012288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=506085376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=127107072, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=943030272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=178184192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=188891136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=14569472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=491732992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=330113024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=398356480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=35921920, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=266612736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=580145152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=316366848, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=762724352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=827146240, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=547999744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=52604928, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=744288256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=389206016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=780058624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=138600448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=568922112, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=947412992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=863789056, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=861249536, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=100061184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=852398080, buflen=4096
fio: pid=160337, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=418398208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=45109248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=866889728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=860798976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=206725120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=248606720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1016901632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=734695424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=123371520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=401780736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=597393408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=365465600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=230748160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=632897536, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=481193984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=670666752, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=588877824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=727089152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=951537664, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=538853376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=309227520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=531943424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=871272448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=495022080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=853266432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=253792256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=360841216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=251990016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=970469376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=50356224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=813268992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=242343936, buflen=4096
fio: pid=160336, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=693460992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=430518272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=229711872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=563814400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=193257472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=79380480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=333225984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=412213248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=33120256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=297811968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=733782016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=771977216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1027985408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=68124672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=781545472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=998039552, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1013321728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1022607360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=184078336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=568000512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=452157440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=175161344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=642146304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=682221568, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=571371520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=67141632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=37384192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=789626880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=604008448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=536186880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1025413120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=72269824, buflen=4096
fio: pid=160345, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=190599168, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=663441408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=490049536, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=794877952, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=233885696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1005789184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1011388416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=909500416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1038233600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=777646080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=626528256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=735145984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=676507648, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=982581248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1060880384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=388255744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=814243840, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=420757504, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=627343360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=442470400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=361467904, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=964497408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=576286720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=581394432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=28442624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=78434304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=784994304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1011179520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=102154240, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=189636608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=48988160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=521666560, buflen=4096
fio: pid=160347, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=455966720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=982298624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=189042688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=827351040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=582787072, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=80031744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=105869312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=555749376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=155660288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1028173824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=651608064, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=94019584, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=562163712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1055100928, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=803667968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=906334208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=426459136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=154656768, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=415268864, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=503599104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=142069760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=313790464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=770019328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=208777216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=874934272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=432349184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=153452544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=909135872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=746696704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=912879616, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=123228160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=411148288, buflen=4096
fio: pid=160341, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=251985920, buflen=4096
fio: pid=160467, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor

Seq-Read-1M-Q8-T1: (groupid=0, jobs=1): err= 0: pid=160241: Sun Aug 20 02:54:25 2023
  read: IOPS=247, BW=248MiB/s (260MB/s)(1024MiB/4135msec)
    slat (usec): min=426, max=328857, avg=3972.90, stdev=22596.07
    clat (usec): min=4, max=335599, avg=25631.80, stdev=53986.45
     lat (usec): min=453, max=517739, avg=29604.70, stdev=59102.60
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    7], 40.00th=[    7], 50.00th=[    8], 60.00th=[    9],
     | 70.00th=[   13], 80.00th=[   28], 90.00th=[   59], 95.00th=[   90],
     | 99.00th=[  330], 99.50th=[  334], 99.90th=[  334], 99.95th=[  338],
     | 99.99th=[  338]
   bw (  KiB/s): min= 1793, max=372736, per=96.84%, avg=245570.62, stdev=129791.86, samples=8
   iops        : min=    1, max=  364, avg=239.62, stdev=126.95, samples=8
  lat (usec)   : 10=0.10%, 500=0.10%
  lat (msec)   : 2=0.20%, 4=1.27%, 10=62.60%, 20=13.77%, 50=7.13%
  lat (msec)   : 100=10.64%, 250=2.15%, 500=2.05%
  cpu          : usr=0.60%, sys=1.33%, ctx=1029, majf=0, minf=2059
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Write-1M-Q8-T1: (groupid=1, jobs=1): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160247: Sun Aug 20 02:54:25 2023
  cpu          : usr=4.85%, sys=0.00%, ctx=18, majf=0, minf=16
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=66.7%, 8=33.3%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Read-1M-Q1-T1: (groupid=2, jobs=1): err= 0: pid=160248: Sun Aug 20 02:54:25 2023
  read: IOPS=282, BW=282MiB/s (296MB/s)(1024MiB/3626msec)
    slat (usec): min=316, max=404103, avg=3474.72, stdev=19094.04
    clat (nsec): min=1948, max=420109, avg=5823.21, stdev=18705.45
     lat (usec): min=318, max=404111, avg=3480.54, stdev=19094.42
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    5], 90.00th=[    7], 95.00th=[    9],
     | 99.00th=[   73], 99.50th=[   93], 99.90th=[  302], 99.95th=[  420],
     | 99.99th=[  420]
   bw (  KiB/s): min=46369, max=376832, per=93.02%, avg=268988.43, stdev=141001.68, samples=7
   iops        : min=   45, max=  368, avg=262.43, stdev=137.69, samples=7
  lat (usec)   : 2=0.10%, 4=73.93%, 10=22.46%, 20=0.88%, 50=1.07%
  lat (usec)   : 100=1.07%, 250=0.29%, 500=0.20%
  cpu          : usr=0.47%, sys=1.30%, ctx=1030, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seq-Write-1M-Q1-T1: (groupid=3, jobs=1): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160253: Sun Aug 20 02:54:25 2023
  cpu          : usr=2.08%, sys=0.00%, ctx=3, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Read-4K-Q32-T16: (groupid=4, jobs=16): err= 0: pid=160255: Sun Aug 20 02:54:25 2023
  read: IOPS=93, BW=373KiB/s (382kB/s)(21.9MiB/60052msec)
    slat (msec): min=51, max=482, avg=170.26, stdev=38.02
    clat (usec): min=7, max=6693.0k, avg=5060592.48, stdev=983003.30
     lat (msec): min=56, max=6868, avg=5230.85, stdev=985.62
    clat percentiles (msec):
     |  1.00th=[  493],  5.00th=[ 3138], 10.00th=[ 4597], 20.00th=[ 4866],
     | 30.00th=[ 5000], 40.00th=[ 5067], 50.00th=[ 5201], 60.00th=[ 5336],
     | 70.00th=[ 5470], 80.00th=[ 5604], 90.00th=[ 5805], 95.00th=[ 6141],
     | 99.00th=[ 6477], 99.50th=[ 6544], 99.90th=[ 6611], 99.95th=[ 6678],
     | 99.99th=[ 6678]
   bw (  KiB/s): min=  135, max=  520, per=98.60%, avg=368.58, stdev= 4.79, samples=1753
   iops        : min=   33, max=  130, avg=91.44, stdev= 1.22, samples=1753
  lat (usec)   : 10=0.14%, 20=0.14%
  lat (msec)   : 100=0.14%, 250=0.18%, 500=0.46%, 750=0.43%, 1000=0.37%
  lat (msec)   : 2000=1.62%, >=2000=96.50%
  cpu          : usr=0.01%, sys=0.02%, ctx=5720, majf=1, minf=674
  IO depths    : 1=0.3%, 2=0.6%, 4=1.1%, 8=2.3%, 16=4.6%, 32=91.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=5603,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Write-4K-Q32-T16: (groupid=5, jobs=16): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160333: Sun Aug 20 02:54:25 2023
  cpu          : usr=0.14%, sys=0.04%, ctx=1030, majf=0, minf=228
  IO depths    : 1=3.1%, 2=6.2%, 4=12.5%, 8=25.0%, 16=50.0%, 32=3.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=66.7%, 8=0.0%, 16=0.0%, 32=33.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Read-4K-Q1-T1: (groupid=6, jobs=1): err= 0: pid=160350: Sun Aug 20 02:54:25 2023
  read: IOPS=12, BW=50.1KiB/s (51.3kB/s)(3008KiB/60027msec)
    slat (msec): min=34, max=132, avg=79.72, stdev=16.86
    clat (usec): min=6, max=275, avg=10.55, stdev=18.51
     lat (msec): min=34, max=132, avg=79.73, stdev=16.86
    clat percentiles (usec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    8],
     | 30.00th=[    8], 40.00th=[    8], 50.00th=[    8], 60.00th=[    9],
     | 70.00th=[    9], 80.00th=[    9], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[  118], 99.50th=[  174], 99.90th=[  277], 99.95th=[  277],
     | 99.99th=[  277]
   bw (  KiB/s): min=   39, max=   64, per=99.78%, avg=50.03, stdev= 5.85, samples=119
   iops        : min=    9, max=   16, avg=12.47, stdev= 1.48, samples=119
  lat (usec)   : 10=96.54%, 20=0.40%, 50=1.46%, 100=0.40%, 250=1.06%
  lat (usec)   : 500=0.13%
  cpu          : usr=0.03%, sys=0.03%, ctx=759, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=752,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Write-4K-Q1-T1: (groupid=7, jobs=1): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160467: Sun Aug 20 02:54:25 2023
  cpu          : usr=0.00%, sys=0.00%, ctx=3, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=248MiB/s (260MB/s), 248MiB/s-248MiB/s (260MB/s-260MB/s), io=1024MiB (1074MB), run=4135-4135msec

Run status group 1 (all jobs):

Run status group 2 (all jobs):
   READ: bw=282MiB/s (296MB/s), 282MiB/s-282MiB/s (296MB/s-296MB/s), io=1024MiB (1074MB), run=3626-3626msec

Run status group 3 (all jobs):

Run status group 4 (all jobs):
   READ: bw=373KiB/s (382kB/s), 373KiB/s-373KiB/s (382kB/s-382kB/s), io=21.9MiB (22.9MB), run=60052-60052msec

Run status group 5 (all jobs):

Run status group 6 (all jobs):
   READ: bw=50.1KiB/s (51.3kB/s), 50.1KiB/s-50.1KiB/s (51.3kB/s-51.3kB/s), io=3008KiB (3080kB), run=60027-60027msec

Run status group 7 (all jobs):

また、fioだとS3のWrite性能が分からなかったので、ddコマンドでWrite性能を確認してみました。

以下コマンドをそれぞれのマウントポイントごとに10回実施した平均が表の結果となります。

ddコマンドによるWrite性能確認

dd if=/dev/zero of=/[EBS、EFS、S3マウントポイント]/.dd-benchmark bs=1M count=1000

EBS dd結果	EFS dd結果	S3 dd結果
415MB/s	316MB/s	303MB/s

ログアウトする際にコンソールが固まる

2023年8月現在、私がテストしたEC2環境ではSystems ManagerのSession ManagerでEC2に接続してS3をマウント→exitでEC2から抜けようとするとコンソールが固まってしまいました。

他のウィンドウから再度EC2にログインしてumountでマウント解除すると固まっていたコンソールも閉じられたので、Mountpoint for Amazon S3が悪さをしているようです。

マウントを実施したコンソール以外のウィンドウは特にexitで抜けても固まることはありませんでした。

また、Session Managerではなく、SSHで接続してS3をマウント→exitであればコンソールが固まることはなかったので、Session Manager＋Mountpoint for Amazon S3の組み合わせが問題を引き起こすようです。

もしSession Managerで操作を行う場合は上記の問題が発生するかと思われるので気をつけましょう。

おわりに

Mountpoint for Amazon S3はそもそもS3なので色々と制約はありますが、シーケンシャルReadに関してはEFSより良い性能が出たのは予想外でした。

また、書き込みについても300MB/s程度出たので、Mountpoint for Amazon S3の特性を理解して使うのであれば、EFSから置き換えることもできるのではないかと思います。

通常のファイルシステムとは勝手が違うので、導入は慎重に行う必要がありますが、うまく要件が合致するようであればS3の利点を活かしつつ、高いパフォーマンスも出すことができる保存領域としてシステムに導入できそうだと感じました。

今回紹介した内容以外の詳細はMountpoint for Amazon S3のGitHubのdocに色々と書かれているので、詳しい情報を知りたい場合は、公式のGitHubを確認してみてください。

• GitHub - awslabs/mountpoint-s3/doc