作ったモデルを評価してみよう
こちらの記事で、llama.cppの標準コマンドで、Sarashina2.2 Vision 3Bが公式に使えることを記事にしました。
この記事は、その続きで、変換後のモデルがどのような特性になったのかを評価してみます。評価は、JMMMUでやってみました。
モデルの公開場所 - Huggingface
こちらに今回利用したggufモデルを登録しています。総合結果も合わせて、 README.mdに記載しています。
評価手順と詳細
前の記事では、VLM/mmproj側は量子化しないようだ、と記載したが、本家のサンプルに --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf というコマンド例の記載がありました。
これにならって、変換したモデルを利用してみることにしました。
各モデルの生成
BF16, F16, Q8_0は、convert_hf_to_gguf.pyで変換可能
Q4_0は、llama-quantizeで変換しないとだめ
嘘か本当かはわからないけど、LLMの量子化枝刈は、ランダム性はないので、一度変換すれば大丈夫の模様
今回の変換
- BF16
- F16
- Q8_0
- Q4_0
- Q4_K_M
BF16
python3 ./convert_hf_to_gguf.py --outtype bf16 --outfile ../models/sara22v/sara22v_bf16.gguf ../cache/hub/models--sbintuitions--sarashina2.2-vision-3b/snapshots/3d4feecd750bb382d01afc6fc8b46035a3155a30/
python3 ./convert_hf_to_gguf.py --outtype bf16 --outfile ../models/sara22v/sara22v_bf16m.gguf --mmproj ../cache/hub/models--sbintuitions--sarashina2.2-vision-3b/snapshots/3d4feecd750bb382d01afc6fc8b46035a3155a30/
F16
python3 ./convert_hf_to_gguf.py --outtype f16 --outfile ../models/sara22v/sara22v_f16.gguf ../cache/hub/models--sbintuitions--sarashina2.2-vision-3b/snapshots/3d4feecd750bb382d01afc6fc8b46035a3155a30/
python3 ./convert_hf_to_gguf.py --outtype f16 --outfile ../models/sara22v/sara22v_f16m.gguf --mmproj ../cache/hub/models--sbintuitions--sarashina2.2-vision-3b/snapshots/3d4feecd750bb382d01afc6fc8b46035a3155a30/
Q8_0
python3 ./convert_hf_to_gguf.py --outtype q8_0 --outfile ../models/sara22v/sara22v_q8.gguf ../cache/hub/models--sbintuitions--sarashina2.2-vision-3b/snapshots/3d4feecd750bb382d01afc6fc8b46035a3155a30/
python3 ./convert_hf_to_gguf.py --outtype q8_0 --outfile ../models/sara22v/sara22v_q8m.gguf --mmproj ../cache/hub/models--sbintuitions--sarashina2.2-vision-3b/snapshots/3d4feecd750bb382d01afc6fc8b46035a3155a30/
Q4_K_M
llama-quantize ./sara22v_bf16.gguf sara22v_q4_k_m.gguf Q4_K_M
llama-quantize ./sara22v_bf16m.gguf sara22v_q4_k_mm.gguf Q4_K_M
変換時に、Q4まで変換できず、Q5やQ8で止まるTensorsがあるようです。一応、変換できたので、そのまま利用します
実行
基本コマンド
export OPENAI_API_BASE="http://localhost:8080/v1"
export OPENAI_API_KEY="dummy"
python3 -m lmms_eval \
--model openai_compatible \
--model_args model_version=(Model Name),azure_openai=False \
--tasks jmmmu \
--batch_size 1 \
--limit 8 \
--log_samples \
--output_path ./logs/
(Model Name)の文字列で、./logs/(Model Name)というフォルダに、出力されます。
BF16
$ time python3 -m lmms_eval --model openai_compatible --model_args model_version=sarav22vm_bf16,azure_openai=False --tasks jmmmu --batch_size 1 --log_samples --output_path ./logs/
(snip)
openai_compatible (model_version=sarav22vm_bf16,azure_openai=False), gen_kwargs: (), limit: None, offset: 0, num_fewshot: None, batch_size: 1
LMMs-Eval: Probing Intelligence in the Real World
> Probing abilities in the real world.
branch: main
commit: v0.6-240-g42296f8f
| Tasks |Filter|n-shot| Metric | |Value | |Stderr|
|---------------------------------------------|------|-----:|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.4000|± |N/A |
| - culture_agnostic |none | |jmmmu_acc|↑ |0.3444|± |N/A |
| - jmmmu_accounting |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_agriculture |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - jmmmu_architecture_and_engineering |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_basic_medical_science |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_biology |none | 0|jmmmu_acc|↑ |0.4333|± |N/A |
| - jmmmu_chemistry |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_clinical_medicine |none | 0|jmmmu_acc|↑ |0.5333|± |N/A |
| - jmmmu_computer_science |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_design |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_diagnostics_and_laboratory_medicine|none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_economics |none | 0|jmmmu_acc|↑ |0.4333|± |N/A |
| - jmmmu_electronics |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_energy_and_power |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_finance |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_manage |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_marketing |none | 0|jmmmu_acc|↑ |0.4333|± |N/A |
| - jmmmu_materials |none | 0|jmmmu_acc|↑ |0.2333|± |N/A |
| - jmmmu_math |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_mechanical_engineering |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_music |none | 0|jmmmu_acc|↑ |0.4333|± |N/A |
| - jmmmu_pharmacy |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - jmmmu_physics |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_psychology |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_public_health |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - culture_specific |none | |jmmmu_acc|↑ |0.4667|± |N/A |
| - jmmmu_japanese_art |none | 0|jmmmu_acc|↑ |0.4667|± |N/A |
| - jmmmu_japanese_heritage |none | 0|jmmmu_acc|↑ |0.4533|± |N/A |
| - jmmmu_japanese_history |none | 0|jmmmu_acc|↑ |0.4200|± |N/A |
| - jmmmu_world_history |none | 0|jmmmu_acc|↑ |0.5267|± |N/A |
Throughput Summary
| Metric | Value | Unit |
|------------------|----------:|--------|
|total_gen_tokens | 16947.0000|tokens |
|total_elapsed_time|252021.2484|seconds |
|avg_speed | 0.0672|tokens/s|
| Groups |Filter|n-shot| Metric | |Value | |Stderr|
|-------------------|------|------|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.4000|± |N/A |
| - culture_agnostic|none | |jmmmu_acc|↑ |0.3444|± |N/A |
| - culture_specific|none | |jmmmu_acc|↑ |0.4667|± |N/A |
real 140m10.238s
user 2m26.145s
sys 0m1.629s
F16
time python3 -m lmms_eval --model openai_compatible --model_args model_version=sarav22vm_f16,azure_openai=False --tasks jmmmu --batch_size 1 --log_samples --output_path ./logs/
(snip)
openai_compatible (model_version=sarav22vm_f16,azure_openai=False), gen_kwargs: (), limit: None, offset: 0, num_fewshot: None, batch_size: 1
LMMs-Eval: Probing Intelligence in the Real World
> Probing abilities in the real world.
branch: main
commit: v0.6-240-g42296f8f
| Tasks |Filter|n-shot| Metric | |Value | |Stderr|
|---------------------------------------------|------|-----:|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.4023|± |N/A |
| - culture_agnostic |none | |jmmmu_acc|↑ |0.3528|± |N/A |
| - jmmmu_accounting |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_agriculture |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - jmmmu_architecture_and_engineering |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_basic_medical_science |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_biology |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_chemistry |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_clinical_medicine |none | 0|jmmmu_acc|↑ |0.5333|± |N/A |
| - jmmmu_computer_science |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_design |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_diagnostics_and_laboratory_medicine|none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_economics |none | 0|jmmmu_acc|↑ |0.4333|± |N/A |
| - jmmmu_electronics |none | 0|jmmmu_acc|↑ |0.4667|± |N/A |
| - jmmmu_energy_and_power |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_finance |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_manage |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_marketing |none | 0|jmmmu_acc|↑ |0.5000|± |N/A |
| - jmmmu_materials |none | 0|jmmmu_acc|↑ |0.2333|± |N/A |
| - jmmmu_math |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_mechanical_engineering |none | 0|jmmmu_acc|↑ |0.2333|± |N/A |
| - jmmmu_music |none | 0|jmmmu_acc|↑ |0.5000|± |N/A |
| - jmmmu_pharmacy |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - jmmmu_physics |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_psychology |none | 0|jmmmu_acc|↑ |0.4333|± |N/A |
| - jmmmu_public_health |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - culture_specific |none | |jmmmu_acc|↑ |0.4617|± |N/A |
| - jmmmu_japanese_art |none | 0|jmmmu_acc|↑ |0.4467|± |N/A |
| - jmmmu_japanese_heritage |none | 0|jmmmu_acc|↑ |0.4600|± |N/A |
| - jmmmu_japanese_history |none | 0|jmmmu_acc|↑ |0.4133|± |N/A |
| - jmmmu_world_history |none | 0|jmmmu_acc|↑ |0.5267|± |N/A |
Throughput Summary
| Metric | Value | Unit |
|------------------|----------:|--------|
|total_gen_tokens | 17215.0000|tokens |
|total_elapsed_time|189932.7277|seconds |
|avg_speed | 0.0906|tokens/s|
| Groups |Filter|n-shot| Metric | |Value | |Stderr|
|-------------------|------|------|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.4023|± |N/A |
| - culture_agnostic|none | |jmmmu_acc|↑ |0.3528|± |N/A |
| - culture_specific|none | |jmmmu_acc|↑ |0.4617|± |N/A |
real 108m48.519s
user 2m25.516s
sys 0m1.740s
Q8_0
time python3 -m lmms_eval --model openai_compatible --model_args model_version=sarav22vm_q8,azure_openai=False --tasks jmmmu --batch_size 1 --log_samples --output_path ./logs/
(snip)
openai_compatible (model_version=sarav22vm_q8,azure_openai=False), gen_kwargs: (), limit: None, offset: 0, num_fewshot: None, batch_size: 1
LMMs-Eval: Probing Intelligence in the Real World
> Better evals lead to better models.
branch: main
commit: v0.6-240-g42296f8f
| Tasks |Filter|n-shot| Metric | |Value | |Stderr|
|---------------------------------------------|------|-----:|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.4000|± |N/A |
| - culture_agnostic |none | |jmmmu_acc|↑ |0.3458|± |N/A |
| - jmmmu_accounting |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_agriculture |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - jmmmu_architecture_and_engineering |none | 0|jmmmu_acc|↑ |0.2000|± |N/A |
| - jmmmu_basic_medical_science |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_biology |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_chemistry |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_clinical_medicine |none | 0|jmmmu_acc|↑ |0.5333|± |N/A |
| - jmmmu_computer_science |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_design |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_diagnostics_and_laboratory_medicine|none | 0|jmmmu_acc|↑ |0.4333|± |N/A |
| - jmmmu_economics |none | 0|jmmmu_acc|↑ |0.4667|± |N/A |
| - jmmmu_electronics |none | 0|jmmmu_acc|↑ |0.4667|± |N/A |
| - jmmmu_energy_and_power |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - jmmmu_finance |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_manage |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - jmmmu_marketing |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_materials |none | 0|jmmmu_acc|↑ |0.2333|± |N/A |
| - jmmmu_math |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_mechanical_engineering |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_music |none | 0|jmmmu_acc|↑ |0.4667|± |N/A |
| - jmmmu_pharmacy |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_physics |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_psychology |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_public_health |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - culture_specific |none | |jmmmu_acc|↑ |0.4650|± |N/A |
| - jmmmu_japanese_art |none | 0|jmmmu_acc|↑ |0.4800|± |N/A |
| - jmmmu_japanese_heritage |none | 0|jmmmu_acc|↑ |0.4733|± |N/A |
| - jmmmu_japanese_history |none | 0|jmmmu_acc|↑ |0.3867|± |N/A |
| - jmmmu_world_history |none | 0|jmmmu_acc|↑ |0.5200|± |N/A |
Throughput Summary
| Metric | Value | Unit |
|------------------|----------:|--------|
|total_gen_tokens | 14232.0000|tokens |
|total_elapsed_time|154425.6469|seconds |
|avg_speed | 0.0922|tokens/s|
| Groups |Filter|n-shot| Metric | |Value | |Stderr|
|-------------------|------|------|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.4000|± |N/A |
| - culture_agnostic|none | |jmmmu_acc|↑ |0.3458|± |N/A |
| - culture_specific|none | |jmmmu_acc|↑ |0.4650|± |N/A |
real 90m57.291s
user 2m26.359s
sys 0m1.528s
Q4_0
time python3 -m lmms_eval --model openai_compatible --model_args model_version=sarav22vm_q4,azure_openai=False --tasks jmmmu --batch_size 1 --log_samples --output_path ./logs/
(snip)
openai_compatible (model_version=sarav22vm_q4,azure_openai=False), gen_kwargs: (), limit: None, offset: 0, num_fewshot: None, batch_size: 1
LMMs-Eval: Probing Intelligence in the Real World
> Good evaluation shapes what we build next.
branch: main
commit: v0.6-240-g42296f8f
| Tasks |Filter|n-shot| Metric | |Value | |Stderr|
|---------------------------------------------|------|-----:|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.3644|± |N/A |
| - culture_agnostic |none | |jmmmu_acc|↑ |0.3125|± |N/A |
| - jmmmu_accounting |none | 0|jmmmu_acc|↑ |0.5000|± |N/A |
| - jmmmu_agriculture |none | 0|jmmmu_acc|↑ |0.2000|± |N/A |
| - jmmmu_architecture_and_engineering |none | 0|jmmmu_acc|↑ |0.1333|± |N/A |
| - jmmmu_basic_medical_science |none | 0|jmmmu_acc|↑ |0.5000|± |N/A |
| - jmmmu_biology |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - jmmmu_chemistry |none | 0|jmmmu_acc|↑ |0.2000|± |N/A |
| - jmmmu_clinical_medicine |none | 0|jmmmu_acc|↑ |0.4333|± |N/A |
| - jmmmu_computer_science |none | 0|jmmmu_acc|↑ |0.2333|± |N/A |
| - jmmmu_design |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_diagnostics_and_laboratory_medicine|none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_economics |none | 0|jmmmu_acc|↑ |0.4667|± |N/A |
| - jmmmu_electronics |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_energy_and_power |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_finance |none | 0|jmmmu_acc|↑ |0.1333|± |N/A |
| - jmmmu_manage |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - jmmmu_marketing |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_materials |none | 0|jmmmu_acc|↑ |0.2333|± |N/A |
| - jmmmu_math |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_mechanical_engineering |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_music |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - jmmmu_pharmacy |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - jmmmu_physics |none | 0|jmmmu_acc|↑ |0.1333|± |N/A |
| - jmmmu_psychology |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - jmmmu_public_health |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - culture_specific |none | |jmmmu_acc|↑ |0.4267|± |N/A |
| - jmmmu_japanese_art |none | 0|jmmmu_acc|↑ |0.4533|± |N/A |
| - jmmmu_japanese_heritage |none | 0|jmmmu_acc|↑ |0.4400|± |N/A |
| - jmmmu_japanese_history |none | 0|jmmmu_acc|↑ |0.3733|± |N/A |
| - jmmmu_world_history |none | 0|jmmmu_acc|↑ |0.4400|± |N/A |
Throughput Summary
| Metric | Value | Unit |
|------------------|----------:|--------|
|total_gen_tokens | 13945.0000|tokens |
|total_elapsed_time|184811.4346|seconds |
|avg_speed | 0.0755|tokens/s|
| Groups |Filter|n-shot| Metric | |Value | |Stderr|
|-------------------|------|------|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.3644|± |N/A |
| - culture_agnostic|none | |jmmmu_acc|↑ |0.3125|± |N/A |
| - culture_specific|none | |jmmmu_acc|↑ |0.4267|± |N/A |
real 112m57.807s
user 2m27.869s
sys 0m1.724s
Q4_K_M
time python3 -m lmms_eval --model openai_compatible --model_args model_version=sarav22vm_q4km,azure_openai=False --tasks jmmmu --batch_size 1 --log_samples --output_path ./logs/
(snip)
openai_compatible (model_version=sarav22vm_q4km,azure_openai=False), gen_kwargs: (), limit: None, offset: 0, num_fewshot: None, batch_size: 1
LMMs-Eval: Probing Intelligence in the Real World
> Measure twice, train once.
branch: main
commit: v0.6-240-g42296f8f
| Tasks |Filter|n-shot| Metric | |Value | |Stderr|
|---------------------------------------------|------|-----:|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.3765|± |N/A |
| - culture_agnostic |none | |jmmmu_acc|↑ |0.3125|± |N/A |
| - jmmmu_accounting |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_agriculture |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_architecture_and_engineering |none | 0|jmmmu_acc|↑ |0.2000|± |N/A |
| - jmmmu_basic_medical_science |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - jmmmu_biology |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - jmmmu_chemistry |none | 0|jmmmu_acc|↑ |0.2000|± |N/A |
| - jmmmu_clinical_medicine |none | 0|jmmmu_acc|↑ |0.5000|± |N/A |
| - jmmmu_computer_science |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - jmmmu_design |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_diagnostics_and_laboratory_medicine|none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_economics |none | 0|jmmmu_acc|↑ |0.4667|± |N/A |
| - jmmmu_electronics |none | 0|jmmmu_acc|↑ |0.3667|± |N/A |
| - jmmmu_energy_and_power |none | 0|jmmmu_acc|↑ |0.2667|± |N/A |
| - jmmmu_finance |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_manage |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_marketing |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_materials |none | 0|jmmmu_acc|↑ |0.1667|± |N/A |
| - jmmmu_math |none | 0|jmmmu_acc|↑ |0.3333|± |N/A |
| - jmmmu_mechanical_engineering |none | 0|jmmmu_acc|↑ |0.4000|± |N/A |
| - jmmmu_music |none | 0|jmmmu_acc|↑ |0.4333|± |N/A |
| - jmmmu_pharmacy |none | 0|jmmmu_acc|↑ |0.3000|± |N/A |
| - jmmmu_physics |none | 0|jmmmu_acc|↑ |0.2333|± |N/A |
| - jmmmu_psychology |none | 0|jmmmu_acc|↑ |0.2000|± |N/A |
| - jmmmu_public_health |none | 0|jmmmu_acc|↑ |0.2333|± |N/A |
| - culture_specific |none | |jmmmu_acc|↑ |0.4533|± |N/A |
| - jmmmu_japanese_art |none | 0|jmmmu_acc|↑ |0.4533|± |N/A |
| - jmmmu_japanese_heritage |none | 0|jmmmu_acc|↑ |0.4733|± |N/A |
| - jmmmu_japanese_history |none | 0|jmmmu_acc|↑ |0.4267|± |N/A |
| - jmmmu_world_history |none | 0|jmmmu_acc|↑ |0.4600|± |N/A |
Throughput Summary
| Metric | Value | Unit |
|------------------|----------:|--------|
|total_gen_tokens | 20922.0000|tokens |
|total_elapsed_time|151628.4773|seconds |
|avg_speed | 0.1380|tokens/s|
| Groups |Filter|n-shot| Metric | |Value | |Stderr|
|-------------------|------|------|---------|---|-----:|---|------|
|jmmmu_all |none | |jmmmu_acc|↑ |0.3765|± |N/A |
| - culture_agnostic|none | |jmmmu_acc|↑ |0.3125|± |N/A |
| - culture_specific|none | |jmmmu_acc|↑ |0.4533|± |N/A |
real 86m7.187s
user 2m26.628s
sys 0m1.600s
結果総括
上の結果は、テキストで見にくいので、各試行を一覧にしました。パット見は、BF16, F16, Q8までは、allにおいて、0.4以上で、0.36, 0.38となっているQ4, Q4_K_Mと比較すると良さそうです。この0.04ほどがどれくらい寄与しているかは不明ですが。また、目的によっては、そもそもフィットしない可能性もあり、このスコア以上に差がある・ない、という状況になるかと思います。
| JMMMU結果 | BF16 | F16 | Q8_0 | Q4_0 | Q4_K_M |
|---|---|---|---|---|---|
| all | 0.4000 | 0.4023 | 0.4000 | 0.3644 | 0.3765 |
| culture_agnostic | 0.3444 | 0.3528 | 0.3458 | 0.3125 | 0.3125 |
| accounting | 0.4000 | 0.4000 | 0.4000 | 0.5000 | 0.4000 |
| agriculture | 0.3667 | 0.3667 | 0.3667 | 0.2000 | 0.3333 |
| architecture_and_engineering | 0.1667 | 0.1667 | 0.2000 | 0.1333 | 0.2000 |
| basic_medical_science | 0.4000 | 0.4000 | 0.4000 | 0.5000 | 0.3667 |
| biology | 0.4333 | 0.4000 | 0.4000 | 0.2667 | 0.3667 |
| chemistry | 0.1667 | 0.1667 | 0.1667 | 0.2000 | 0.2000 |
| clinical_medicine | 0.5333 | 0.5333 | 0.5333 | 0.4333 | 0.5000 |
| computer_science | 0.3333 | 0.3333 | 0.3000 | 0.2333 | 0.2667 |
| design | 0.4000 | 0.4000 | 0.4000 | 0.4000 | 0.3333 |
| diagnostics_and_laboratory_medicine | 0.4000 | 0.4000 | 0.4333 | 0.3333 | 0.3000 |
| economics | 0.4333 | 0.4333 | 0.4667 | 0.4667 | 0.4667 |
| electronics | 0.4000 | 0.4667 | 0.4667 | 0.4000 | 0.3667 |
| energy_and_power | 0.3000 | 0.3333 | 0.2667 | 0.3000 | 0.2667 |
| finance | 0.1667 | 0.1667 | 0.1667 | 0.1333 | 0.1667 |
| manage | 0.3333 | 0.3333 | 0.3667 | 0.2667 | 0.3333 |
| marketing | 0.4333 | 0.5000 | 0.4000 | 0.3333 | 0.3333 |
| materials | 0.2333 | 0.2333 | 0.2333 | 0.2333 | 0.1667 |
| math | 0.4000 | 0.4000 | 0.3333 | 0.3333 | 0.3333 |
| mechanical_engineering | 0.3000 | 0.2333 | 0.3000 | 0.3333 | 0.4000 |
| music | 0.4333 | 0.5000 | 0.4667 | 0.3667 | 0.4333 |
| pharmacy | 0.2667 | 0.2667 | 0.3000 | 0.2667 | 0.3000 |
| physics | 0.3000 | 0.3333 | 0.3000 | 0.1333 | 0.2333 |
| psychology | 0.4000 | 0.4333 | 0.3333 | 0.3667 | 0.2000 |
| public_health | 0.2667 | 0.2667 | 0.3000 | 0.3667 | 0.2333 |
| culture_specific | 0.4667 | 0.4617 | 0.4650 | 0.4267 | 0.4533 |
| japanese_art | 0.4667 | 0.4467 | 0.4800 | 0.4533 | 0.4533 |
| japanese_heritage | 0.4533 | 0.4600 | 0.4733 | 0.4400 | 0.4733 |
| japanese_history | 0.4200 | 0.4133 | 0.3867 | 0.3733 | 0.4267 |
| world_history | 0.5267 | 0.5267 | 0.5200 | 0.4400 | 0.4600 |
timeで計測している値と乖離があるので、参考程度です。また、トライによっては、リトライが走っているので、その分余計にトークンを消費している可能性があります。
また、自作サーバでの運用なので、遅いです。ただ、どの回も、同じハードウェアで行っているので、環境さはそれほどないはずです。歯切れが悪いのは、裏でコマンドを叩いていたりすると、速度がゆらぎが発生していると思います。
| Metric | BF16 | F16 | Q8_0 | Q4_0 | Q4_K_M |
|---|---|---|---|---|---|
| total_gen_tokens (tokens) | 16947 | 17215 | 14232 | 13945 | 20922 |
| total_elapsed_time (seconds) | 252021.2484 | 189932.7277 | 154425.6469 | 184811.4346 | 151628.4773 |
| avg_speed (tokens/s) | 0.0672 | 0.0906 | 0.0922 | 0.0755 | 0.1380 |
おまけ
ハードウェアは次のとおりです。
- 13th Intel i7
- Mem 64G
- RTX3060 (12G)
KVM上で、Ubuntu22.04を動かして試験しました。
今日のところは、ここまで。
おしまい