0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

動画生成AIのarenaスコア

0
Posted at

動画生成AI(text2video)の2026/09/30の上位20位の勝率、および試合数は以下の通りだった。

勝率(Tie除く)

image.png

試合数(Tie除く)

image.png

ここで勝率の表を見ていて自分はある引っ掛かりを覚えた。
というのも勝率が85%の場合、Elo差は通常300程度あるはずなのに、
$\frac{1}{1+10^{-300/400}}=0.8490$
gemini-omni-1.1-flashが1516、wan2.7-t2vが1337でElo差が179しかなかったからである。

image.png

ここでchatgptに前述の図からOCRしてもらって上位20位モデルのみのBradley-Terry modelのElo計算してもらった。
左が上位20モデルのみで求めたElo差で、右がオリジナルのElo差である。
こうみるとオリジナルの上位20位モデルのElo差は正しくないように見える。

なお、Elo差0付近で最もずれているのはveo-3.1-audio-1080p,veo-3-audioの組み合わせでほぼ同程度のEloであるのに、試合数は23試合しかなく勝率0.74である。
image.png

また、両方の平均Eloを1200に合わせた際の結果が以下である。
この場合、seedance-2.0の順位はわずかに下がり、wan3.0の順位はわずかに上がる。

====================================================================================================
Calculated Elo vs Original Elo (both normalized to mean 1200)
====================================================================================================
Model                                     Calculated    Original  Difference
gemini-omni-1.1-flash                    1359.70 (1360) 1300.50 (1516)      +59.20
gemini-omni-flash                        1357.77 (1358) 1297.50 (1513)      +60.27
flux-3-video                             1345.21 (1345) 1277.50 (1493)      +67.71
grok-imagine-video-1.5-agent             1303.74 (1304) 1276.50 (1492)      +27.24
dreamina-seedance-2.0-720p               1277.08 (1277) 1263.50 (1479)      +13.58
wan3.0                                   1310.13 (1310) 1260.50 (1476)      +49.63
dreamina-seedance-2.5-720p               1297.03 (1297) 1258.50 (1474)      +38.53
minimax-h3                               1270.09 (1270) 1244.50 (1460)      +25.59
muse-video                               1262.68 (1263) 1240.50 (1456)      +22.18
happyhorse-1.0                           1218.02 (1218) 1211.50 (1427)       +6.52
sora-2-pro                               1120.33 (1120) 1152.50 (1368)      -32.17
veo-3.1-audio                            1128.26 (1128) 1148.50 (1364)      -20.24
veo-3.1-audio-1080p                      1095.19 (1095) 1146.50 (1362)      -51.31
veo-3.1-fast-audio                       1112.12 (1112) 1146.50 (1362)      -34.38
veo-3.1-fast-audio-1080p                 1086.09 (1086) 1142.50 (1358)      -56.41
veo-3-fast-audio                         1085.78 (1086) 1132.50 (1348)      -46.72
sora-2                                   1100.76 (1101) 1127.50 (1343)      -26.74
grok-imagine-video-720p                  1099.53 (1100) 1126.50 (1342)      -26.97
veo-3-audio                              1092.58 (1093) 1124.50 (1340)      -31.92
wan2.7-t2v                               1077.92 (1078) 1121.50 (1337)      -43.58

何故ずれるのか

これは今回したElo計算が全体データの一部(上位20モデル)でしかないためである。
表示されるデータ以外のデータ(もっと下位のモデルとの対戦結果)を使うとその勝率に合わせようとするためそれに引きずられ、(全体としては合うが)上位だけの対戦結果だとずれる。

あるいは実際のAIモデルの実力分布が単純な山形になってないため、一本のElo軸への押し込みに失敗しているのかもしれない。

また、AIモデルは常に自分の実力を発揮できるわけではない。各モデルごとに得意分野の違いが存在したり、下位モデルのラッキーパンチと上位モデルのやらかしが重なってしまった場合がある。あるいは評価者の目が肥えて特定モデルに違和感を感じるようになってしまったり、評価者側の国籍によって好む顔が違ったりすることもある。

まあ、いずれにしても上位モデル間のLeaderboardのElo差は小さく見えるが実際には上位間のElo差はもう少し広がるようだ。

参考:コード

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm


# ============================================================
# データ読み込み
# ============================================================

path = 'arena.csv'
df = pd.read_csv(path)

path2 = 'arena_win.csv'
df2 = pd.read_csv(path2)

path3 = 'arena_elo_original.csv'
df_original = pd.read_csv(path3)


match_num = df.to_numpy()[:, 1:].astype(np.float32)
win_rate = df2.to_numpy()[:, 1:].astype(np.float32)
win_num = match_num * win_rate

models = df.columns[1:].tolist()

n = len(match_num)


# ============================================================
# Elo計算
# ============================================================

Elo_pred = np.ones(n) * 1200
a = np.log(10)

Elo_list = []

for ite in range(2000):

    diff = Elo_pred[:, None] - Elo_pred[None, :]

    pred = 1 / (1 + np.exp(-a * diff / 400))

    g = np.sum(win_num - match_num * pred, axis=1)
    h = -np.sum(match_num * pred * (1 - pred), axis=1)

    delta = -g / (h + 1e-8)

    Elo_pred += delta

    # 平均を1200に固定
    Elo_pred -= (np.mean(Elo_pred) - 1200)

    Elo_list.append(np.copy(Elo_pred))


Elo_list = np.asarray(Elo_list)

print('Elo_pred =')
for model, elo in zip(models, Elo_pred):
    print(f'{model:40s} {elo - 1200:8.2f}')


# ============================================================
# Original Eloを読み込み
# ============================================================

original_dict = dict(
    zip(
        df_original['Model'],
        df_original['arena_elo']
    )
)

# arena.csv に存在するモデルだけ取り出す
original_elo = np.array([
    original_dict[model]
    for model in models
])

# ============================================================
# Calculated Elo / Original Elo を平均1200に補正
# ============================================================

# Calculated Elo
calculated_elo_original = Elo_pred.copy()
calculated_elo = (
    calculated_elo_original
    - np.mean(calculated_elo_original)
    + 1200
)

# Original Elo
original_elo_original = np.array([
    original_dict[model]
    for model in models
])

original_elo = (
    original_elo_original
    - np.mean(original_elo_original)
    + 1200
)


# ============================================================
# Elo差の比較
# ============================================================

elo_difference = calculated_elo - original_elo


print()
print('=' * 100)
print('Calculated Elo vs Original Elo (both normalized to mean 1200)')
print('=' * 100)

print(
    f'{"Model":40s}'
    f'{"Calculated":>12s}'
    f'{"Original":>12s}'
    f'{"Difference":>12s}'
)

for model, calc, orig, diff, calc_raw, orig_raw in zip(
    models,
    calculated_elo,
    original_elo,
    elo_difference,
    calculated_elo_original,
    original_elo_original
):
    print(
        f'{model:40s}'
        f'{calc:8.2f} ({calc_raw:.0f})'
        f'{orig:8.2f} ({orig_raw:.0f})'
        f'{diff:+12.2f}'
    )

# ============================================================
# ペアごとのデータを作成
# ============================================================

elo_diff_calculated = []
elo_diff_original = []
observed_win_rate = []
weights = []

for i in range(n):
    for j in range(n):

        if i == j:
            continue

        if match_num[i, j] <= 0:
            continue

        # 今回計算したElo
        diff_calc = Elo_pred[i] - Elo_pred[j]

        # Original Elo
        diff_orig = original_elo[i] - original_elo[j]

        elo_diff_calculated.append(diff_calc)
        elo_diff_original.append(diff_orig)

        observed_win_rate.append(win_rate[i, j])
        weights.append(match_num[i, j])


elo_diff_calculated = np.asarray(elo_diff_calculated)
elo_diff_original = np.asarray(elo_diff_original)
observed_win_rate = np.asarray(observed_win_rate)
weights = np.asarray(weights)


# ============================================================
# 両グラフで共通のX軸範囲
# ============================================================

xmin = min(
    elo_diff_calculated.min(),
    elo_diff_original.min()
)

xmax = max(
    elo_diff_calculated.max(),
    elo_diff_original.max()
)

# 少し余白を追加
margin = (xmax - xmin) * 0.03

xmin -= margin
xmax += margin


# ============================================================
# グラフ
# ============================================================
# ============================================================
# グラフ
# ============================================================

fig = plt.figure(figsize=(18, 7))

# 3列目をカラーバー専用にする
gs = fig.add_gridspec(
    1, 3,
    width_ratios=[1, 1, 0.045],
    wspace=0.08
)

axes = [
    fig.add_subplot(gs[0, 0]),
    fig.add_subplot(gs[0, 1])
]

cax = fig.add_subplot(gs[0, 2])


# ============================================================
# 左:今回計算したElo
# ============================================================

sc1 = axes[0].scatter(
    elo_diff_calculated,
    observed_win_rate,
    c=weights,
    cmap='viridis',
    norm=LogNorm(
        vmin=weights.min(),
        vmax=weights.max()
    ),
    alpha=0.7,
    s=30,
)

x = np.linspace(xmin, xmax, 500)

y = 1 / (
    1 + np.exp(-np.log(10) * x / 400)
)

axes[0].plot(
    x,
    y,
    'k--',
    linewidth=2,
    label='Theoretical Elo curve'
)

axes[0].set_title('Calculated Elo')
axes[0].set_xlabel('Elo difference')
axes[0].set_ylabel('Observed win rate')

axes[0].set_xlim(xmin, xmax)
axes[0].set_ylim(0, 1)

axes[0].grid(alpha=0.3)
axes[0].legend()


# ============================================================
# 右:Original Elo
# ============================================================

sc2 = axes[1].scatter(
    elo_diff_original,
    observed_win_rate,
    c=weights,
    cmap='viridis',
    norm=LogNorm(
        vmin=weights.min(),
        vmax=weights.max()
    ),
    alpha=0.7,
    s=30,
)

axes[1].plot(
    x,
    y,
    'k--',
    linewidth=2,
    label='Theoretical Elo curve'
)

axes[1].set_title('Original Elo')
axes[1].set_xlabel('Elo difference')

axes[1].set_xlim(xmin, xmax)
axes[1].set_ylim(0, 1)

axes[1].grid(alpha=0.3)
axes[1].legend()


# ============================================================
# カラーバーを右端に独立配置
# ============================================================

cbar = fig.colorbar(
    sc1,
    cax=cax
)

cbar.set_label('Number of matches')


# ============================================================
# タイトル
# ============================================================

fig.suptitle(
    'Elo difference vs observed win rate',
    fontsize=16
)

plt.show()
0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?