昨日の記事の手法を「HPLT Bert for Belarusian」とUD_Belarusian-HSEに適用して、ベラルーシ語係り受け解析モデルltgbert-base-belarusian-ud-goeswithを試作してみた。be_hse-ud-test.conlluによるベンチマーク・プログラム(Google Colaboratory GPU版)は、過去に作ったベラルーシ語モデルも含めると、こんな感じ。
!pip install esupar
models=["KoichiYasuoka/ltgbert-base-belarusian-ud-goeswith","KoichiYasuoka/deberta-base-belarusian-upos","KoichiYasuoka/deberta-base-belarusian-ud-goeswith","KoichiYasuoka/roberta-small-belarusian-upos"]
import os,sys,subprocess,esupar,deplacy
from transformers import pipeline
url="https://github.com/UniversalDependencies/UD_Belarusian-HSE"
d=os.path.basename(url)
os.system(f"test -d {d} || git clone --depth=1 {url}")
os.system("for F in train dev test ; do cp "+d+"/*-$F.conllu $F.conllu ; done")
url="https://universaldependencies.org/conll18/conll18_ud_eval.py"
c=os.path.basename(url)
os.system(f"test -f {c} || curl -LO {url}")
with open("test.conllu","r",encoding="utf-8") as r:
s=[t[8:].strip() for t in r if t.startswith("# text =")]
for mdl in models:
if mdl.endswith("-upos"):
nlp=esupar.load(mdl)
else:
nlp=pipeline("universal-dependencies",mdl,trust_remote_code=True,aggregation_strategy="simple",device=0)
with open("result.conllu","w",encoding="utf-8") as w:
for t in s:
print(deplacy.to_conllu(nlp(t)).strip(),end="\n\n",file=w)
p=subprocess.run([sys.executable,c,"-v","test.conllu","result.conllu"],encoding="utf-8",stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
print(f"\n*** {mdl}",p.stdout,sep="\n",flush=True)
私(安岡孝一)の手元では、以下の結果が出力された。
*** KoichiYasuoka/ltgbert-base-belarusian-ud-goeswith
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.57 | 99.47 | 99.52 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.57 | 99.47 | 99.52 |
UPOS | 98.37 | 98.27 | 98.32 | 98.80
XPOS | 0.00 | 0.00 | 0.00 | 0.00
UFeats | 95.43 | 95.33 | 95.38 | 95.84
AllTags | 0.00 | 0.00 | 0.00 | 0.00
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 92.37 | 92.28 | 92.33 | 92.77
LAS | 90.39 | 90.30 | 90.35 | 90.79
CLAS | 88.58 | 88.38 | 88.48 | 88.84
MLAS | 82.88 | 82.69 | 82.78 | 83.12
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/deberta-base-belarusian-upos
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.76 | 99.89 | 99.83 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.76 | 99.89 | 99.83 |
UPOS | 98.99 | 99.12 | 99.06 | 99.23
XPOS | 0.00 | 0.00 | 0.00 | 0.00
UFeats | 97.17 | 97.29 | 97.23 | 97.40
AllTags | 0.00 | 0.00 | 0.00 | 0.00
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 88.94 | 89.05 | 88.99 | 89.15
LAS | 86.73 | 86.84 | 86.78 | 86.93
CLAS | 84.24 | 84.15 | 84.20 | 84.27
MLAS | 80.28 | 80.19 | 80.24 | 80.31
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/deberta-base-belarusian-ud-goeswith
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.47 | 99.66 | 99.56 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.47 | 99.66 | 99.56 |
UPOS | 97.92 | 98.11 | 98.01 | 98.44
XPOS | 0.00 | 0.00 | 0.00 | 0.00
UFeats | 94.78 | 94.96 | 94.87 | 95.28
AllTags | 0.00 | 0.00 | 0.00 | 0.00
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 89.32 | 89.49 | 89.41 | 89.80
LAS | 86.89 | 87.05 | 86.97 | 87.35
CLAS | 84.51 | 84.28 | 84.40 | 84.62
MLAS | 78.47 | 78.26 | 78.37 | 78.58
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/roberta-small-belarusian-upos
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.45 | 99.50 | 99.48 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.45 | 99.50 | 99.48 |
UPOS | 94.66 | 94.71 | 94.68 | 95.18
XPOS | 0.00 | 0.00 | 0.00 | 0.00
UFeats | 83.66 | 83.70 | 83.68 | 84.12
AllTags | 0.00 | 0.00 | 0.00 | 0.00
Lemmas | 0.00 | 0.00 | 0.00 | 0.00
UAS | 87.21 | 87.25 | 87.23 | 87.69
LAS | 84.80 | 84.84 | 84.82 | 85.27
CLAS | 81.93 | 81.90 | 81.91 | 82.37
MLAS | 62.22 | 62.20 | 62.21 | 62.56
BLEX | 0.00 | 0.00 | 0.00 | 0.00
ltgbert-base-belarusian-ud-goeswithがダントツで、UPOS/LAS/MLASが98.32/90.35/82.78だったりする。ただ、HPLT Deliverable 4.1『First language models trained』(2024年2月29日)のTable 3.5によれば、ベラルーシ語のLASは91.1まで行ったらしいので、もうちょっと改善の余地があるのかな。