3月30日の記事の続きだが、ModernBERT-base-Turkish-uncased-mlmのトークナイザを入れ替えつつ、トルコ語品詞付与・係り受け解析モデルmodernbert-base-turkish-ud-goeswithとmodernbert-base-turkish-ud-embedsを試作してみた。ただ、どうにも解析精度が上がらない。UD_Turkish-Kenetのtr_kenet-ud-test.conlluと、UD_Turkish-Pennのtr_penn-ud-test.conlluで、他のトルコ語モデルといっしょに解析精度を測ってみよう。Google Colaboratoryだと、こんな感じ。
!pip install transformers triton
models=[
"KoichiYasuoka/bert-base-turkish-ud-goeswith",
"KoichiYasuoka/bert-large-turkish-ud-goeswith",
"KoichiYasuoka/roberta-base-turkish-ud-goeswith",
"KoichiYasuoka/ltgbert-base-turkish-ud-goeswith",
"KoichiYasuoka/modernbert-base-turkish-ud-goeswith",
"KoichiYasuoka/modernbert-base-turkish-ud-embeds"
]
import os,sys,subprocess
url="https://github.com/UniversalDependencies/UD_Turkish-"
tests=["Kenet","Penn"]
for t in tests:
u=url+t+"/raw/refs/heads/master/tr_"+t.lower()+"-ud-test.conllu"
f=os.path.basename(u)
os.system(f"test -f {f} || curl -LO {u}")
url="https://universaldependencies.org/conll18/conll18_ud_eval.py"
c=os.path.basename(url)
os.system(f"test -f {c} || curl -LO {url}")
for mdl in models:
from transformers import pipeline
nlp=pipeline("universal-dependencies",mdl,trust_remote_code=True,aggregation_strategy="simple")
for f in tests:
with open(f"tr_{f.lower()}-ud-test.conllu","r",encoding="utf-8") as r:
s=[t[8:].strip() for t in r if t.startswith("# text =")]
with open(f,"w",encoding="utf-8") as w:
for t in s:
w.write(nlp(t))
os.system(f"mkdir -p result/{mdl}")
with open(f"result/{mdl}/result.txt","w",encoding="utf-8") as w:
for f in tests:
p=subprocess.run([sys.executable,c,"-v",f"tr_{f.lower()}-ud-test.conllu",f],encoding="utf-8",stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
print(f"\n*** {mdl} ({f})",p.stdout,sep="\n",file=w)
!( cd result && cat `find {" ".join(models)} -name result.txt` )
私(安岡孝一)の手元では、以下の結果が出力された。
*** KoichiYasuoka/bert-base-turkish-ud-goeswith (Kenet)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.74 | 99.47 | 99.60 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.74 | 99.47 | 99.60 |
UPOS | 93.53 | 93.28 | 93.41 | 93.78
XPOS | 99.74 | 99.47 | 99.60 | 100.00
UFeats | 91.33 | 91.09 | 91.21 | 91.58
AllTags | 90.51 | 90.26 | 90.38 | 90.75
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 85.15 | 84.92 | 85.03 | 85.37
LAS | 72.98 | 72.78 | 72.88 | 73.17
CLAS | 73.61 | 72.49 | 73.05 | 72.75
MLAS | 64.16 | 63.18 | 63.66 | 63.40
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/bert-base-turkish-ud-goeswith (Penn)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 98.26 | 98.12 | 98.19 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 98.26 | 98.12 | 98.19 |
UPOS | 94.34 | 94.21 | 94.27 | 96.01
XPOS | 98.26 | 98.12 | 98.19 | 100.00
UFeats | 93.06 | 92.93 | 93.00 | 94.71
AllTags | 92.22 | 92.09 | 92.15 | 93.85
Lemmas | 0.01 | 0.01 | 0.01 | 0.01
UAS | 84.29 | 84.17 | 84.23 | 85.79
LAS | 72.13 | 72.03 | 72.08 | 73.41
CLAS | 69.96 | 68.90 | 69.43 | 70.17
MLAS | 62.74 | 61.79 | 62.26 | 62.92
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/bert-large-turkish-ud-goeswith (Kenet)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.94 | 99.88 | 99.91 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.94 | 99.88 | 99.91 |
UPOS | 93.20 | 93.14 | 93.17 | 93.25
XPOS | 99.94 | 99.88 | 99.91 | 100.00
UFeats | 90.85 | 90.79 | 90.82 | 90.90
AllTags | 89.92 | 89.87 | 89.90 | 89.98
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 83.59 | 83.54 | 83.56 | 83.64
LAS | 71.00 | 70.95 | 70.98 | 71.04
CLAS | 71.17 | 70.30 | 70.73 | 70.36
MLAS | 61.55 | 60.80 | 61.17 | 60.85
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/bert-large-turkish-ud-goeswith (Penn)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 98.58 | 98.98 | 98.78 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 98.58 | 98.98 | 98.78 |
UPOS | 94.58 | 94.96 | 94.77 | 95.94
XPOS | 98.58 | 98.98 | 98.78 | 100.00
UFeats | 93.02 | 93.40 | 93.21 | 94.36
AllTags | 92.16 | 92.54 | 92.35 | 93.48
Lemmas | 0.01 | 0.01 | 0.01 | 0.01
UAS | 84.47 | 84.81 | 84.64 | 85.68
LAS | 71.91 | 72.20 | 72.05 | 72.94
CLAS | 69.21 | 68.47 | 68.83 | 69.32
MLAS | 61.79 | 61.13 | 61.46 | 61.90
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/roberta-base-turkish-ud-goeswith (Kenet)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 92.43 | 94.91 | 93.66 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 92.43 | 94.91 | 93.66 |
UPOS | 76.73 | 78.78 | 77.74 | 83.01
XPOS | 92.43 | 94.91 | 93.66 | 100.00
UFeats | 66.90 | 68.69 | 67.79 | 72.38
AllTags | 63.94 | 65.66 | 64.79 | 69.18
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 58.55 | 60.12 | 59.33 | 63.35
LAS | 43.41 | 44.57 | 43.98 | 46.96
CLAS | 46.99 | 45.58 | 46.27 | 48.51
MLAS | 32.28 | 31.31 | 31.79 | 33.32
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/roberta-base-turkish-ud-goeswith (Penn)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 89.92 | 87.28 | 88.58 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 89.92 | 87.28 | 88.58 |
UPOS | 76.18 | 73.94 | 75.04 | 84.72
XPOS | 89.92 | 87.28 | 88.58 | 100.00
UFeats | 69.91 | 67.86 | 68.87 | 77.75
AllTags | 67.39 | 65.41 | 66.39 | 74.95
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 62.17 | 60.35 | 61.25 | 69.14
LAS | 47.23 | 45.84 | 46.53 | 52.53
CLAS | 49.32 | 45.71 | 47.44 | 53.03
MLAS | 36.48 | 33.81 | 35.09 | 39.22
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/ltgbert-base-turkish-ud-goeswith (Kenet)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.84 | 99.89 | 99.87 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.84 | 99.89 | 99.87 |
UPOS | 91.09 | 91.13 | 91.11 | 91.23
XPOS | 99.84 | 99.89 | 99.87 | 100.00
UFeats | 86.25 | 86.29 | 86.27 | 86.39
AllTags | 84.98 | 85.01 | 85.00 | 85.11
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 78.68 | 78.71 | 78.69 | 78.80
LAS | 65.86 | 65.89 | 65.87 | 65.96
CLAS | 64.98 | 64.13 | 64.55 | 64.23
MLAS | 53.86 | 53.16 | 53.51 | 53.24
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/ltgbert-base-turkish-ud-goeswith (Penn)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.63 | 99.74 | 99.69 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.63 | 99.74 | 99.69 |
UPOS | 94.74 | 94.84 | 94.79 | 95.09
XPOS | 99.63 | 99.74 | 99.69 | 100.00
UFeats | 92.24 | 92.35 | 92.30 | 92.59
AllTags | 91.18 | 91.28 | 91.23 | 91.52
Lemmas | 0.01 | 0.01 | 0.01 | 0.01
UAS | 82.39 | 82.48 | 82.44 | 82.70
LAS | 69.58 | 69.65 | 69.61 | 69.83
CLAS | 66.68 | 66.04 | 66.36 | 66.26
MLAS | 57.84 | 57.28 | 57.56 | 57.47
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/modernbert-base-turkish-ud-goeswith (Kenet)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.99 | 99.97 | 99.98 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.99 | 99.97 | 99.98 |
UPOS | 92.83 | 92.82 | 92.82 | 92.84
XPOS | 99.99 | 99.97 | 99.98 | 100.00
UFeats | 90.33 | 90.31 | 90.32 | 90.34
AllTags | 89.43 | 89.41 | 89.42 | 89.44
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 84.07 | 84.05 | 84.06 | 84.08
LAS | 70.32 | 70.31 | 70.32 | 70.33
CLAS | 70.13 | 69.49 | 69.81 | 69.51
MLAS | 60.31 | 59.76 | 60.03 | 59.78
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/modernbert-base-turkish-ud-goeswith (Penn)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 98.32 | 98.72 | 98.52 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 98.32 | 98.72 | 98.52 |
UPOS | 93.73 | 94.11 | 93.92 | 95.33
XPOS | 98.32 | 98.72 | 98.52 | 100.00
UFeats | 92.29 | 92.65 | 92.47 | 93.86
AllTags | 91.33 | 91.69 | 91.51 | 92.88
Lemmas | 0.01 | 0.01 | 0.01 | 0.01
UAS | 83.53 | 83.87 | 83.70 | 84.96
LAS | 70.81 | 71.10 | 70.95 | 72.02
CLAS | 67.99 | 67.48 | 67.74 | 68.48
MLAS | 60.24 | 59.79 | 60.01 | 60.68
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/modernbert-base-turkish-ud-embeds (Kenet)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.98 | 99.98 | 99.98 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.98 | 99.98 | 99.98 |
UPOS | 92.40 | 92.40 | 92.40 | 92.42
XPOS | 99.98 | 99.98 | 99.98 | 100.00
UFeats | 89.40 | 89.40 | 89.40 | 89.42
AllTags | 88.48 | 88.48 | 88.48 | 88.50
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 82.26 | 82.26 | 82.26 | 82.27
LAS | 68.52 | 68.52 | 68.52 | 68.54
CLAS | 67.78 | 67.23 | 67.50 | 67.24
MLAS | 58.43 | 57.95 | 58.19 | 57.97
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/modernbert-base-turkish-ud-embeds (Penn)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 98.36 | 98.93 | 98.64 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 98.36 | 98.93 | 98.64 |
UPOS | 93.72 | 94.26 | 93.99 | 95.28
XPOS | 98.36 | 98.93 | 98.64 | 100.00
UFeats | 91.84 | 92.37 | 92.10 | 93.37
AllTags | 90.95 | 91.47 | 91.21 | 92.46
Lemmas | 0.01 | 0.01 | 0.01 | 0.01
UAS | 82.29 | 82.76 | 82.52 | 83.66
LAS | 69.30 | 69.70 | 69.50 | 70.46
CLAS | 67.24 | 66.81 | 67.03 | 67.70
MLAS | 59.58 | 59.20 | 59.39 | 59.99
BLEX | 0.00 | 0.00 | 0.00 | 0.00
UPOS/LAS/MLASを表にしてみよう。
| tr_kenet-ud-test.conllu | tr_penn-ud-test.conllu | |
|---|---|---|
| bert-base-turkish-ud-goeswith | 93.41/72.88/63.66 | 94.27/72.08/62.26 |
| bert-large-turkish-ud-goeswith | 93.17/70.98/61.17 | 94.77/72.05/61.46 |
| roberta-base-turkish-ud-goeswith | 77.74/43.98/31.79 | 75.04/46.53/35.09 |
| ltgbert-base-turkish-ud-goeswith | 91.11/65.87/53.51 | 94.79/69.61/57.56 |
| modernbert-base-turkish-ud-goeswith | 92.82/70.32/60.03 | 93.92/70.95/60.01 |
| modernbert-base-turkish-ud-embeds | 92.40/68.52/58.19 | 93.99/69.50/59.39 |
bert-base-turkish-ud-goeswithがトップではあるものの、どのモデルも、まだまだ伸びしろがありそうだ。さて、どういうチューニングがトルコ語に向いてるのかな。