昨日の記事の続きだが、Panyut Sriwirote, Wei Qi Leong, Charin Polpanumas, Santhawat Thanyawong, William Chandra Tjhi, Wirote Aroonmanakun, Attapol T. Rutherford『Thai Universal Dependency Treebank』のtest.conlluで、タイ語係り受け解析esuparモデルの「精度」を測ってみた。Google Colaboratoryだと、こんな感じ。
!pip install esupar
models=["KoichiYasuoka/camembert-thai-base-upos","KoichiYasuoka/deberta-base-thai-upos","KoichiYasuoka/roberta-base-thai-spm-upos","KoichiYasuoka/roberta-base-thai-syllable-upos","KoichiYasuoka/roberta-base-thai-char-upos","KoichiYasuoka/bert-base-thai-upos"]
import os,sys,subprocess,esupar
url="https://github.com/nlp-chula/TUD"
f=os.path.join(os.path.basename(url),"TUD","test.conllu")
os.system(f"test -f {f} || git clone --depth=1 {url}")
url="https://universaldependencies.org/conll18/conll18_ud_eval.py"
c=os.path.basename(url)
os.system(f"test -f {c} || curl -LO {url}")
with open(f,"r",encoding="utf-8") as r:
s=r.read()
for mdl in models:
nlp=esupar.load(mdl)
with open("result.conllu","w",encoding="utf-8") as w:
for t in s.split("\n"):
if t.startswith("# text = "):
print(nlp(t[9:]),file=w)
p=subprocess.run([sys.executable,c,"-v",f,"result.conllu"],encoding="utf-8",stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
print(f"\n*** {mdl}",p.stdout,sep="\n",flush=True)
私(安岡孝一)の手元では、以下の結果が出力された。
*** KoichiYasuoka/camembert-thai-base-upos
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 92.58 | 89.80 | 91.17 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 92.58 | 89.80 | 91.17 |
UPOS | 80.92 | 78.48 | 79.68 | 87.40
XPOS | 92.58 | 89.80 | 91.17 | 100.00
UFeats | 89.18 | 86.50 | 87.82 | 96.33
AllTags | 77.76 | 75.43 | 76.58 | 84.00
Lemmas | 92.58 | 89.80 | 91.17 | 100.00
UAS | 74.56 | 72.32 | 73.42 | 80.53
LAS | 63.53 | 61.62 | 62.56 | 68.62
CLAS | 60.47 | 56.63 | 58.49 | 64.39
MLAS | 48.82 | 45.73 | 47.22 | 51.99
BLEX | 60.47 | 56.63 | 58.49 | 64.39
*** KoichiYasuoka/deberta-base-thai-upos
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 91.07 | 88.81 | 89.92 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 91.07 | 88.81 | 89.92 |
UPOS | 75.80 | 73.92 | 74.85 | 83.23
XPOS | 91.07 | 88.81 | 89.92 | 100.00
UFeats | 87.75 | 85.57 | 86.64 | 96.35
AllTags | 72.70 | 70.90 | 71.79 | 79.83
Lemmas | 91.07 | 88.81 | 89.92 | 100.00
UAS | 71.06 | 69.30 | 70.17 | 78.03
LAS | 60.16 | 58.66 | 59.40 | 66.06
CLAS | 56.75 | 53.51 | 55.08 | 61.86
MLAS | 43.62 | 41.13 | 42.34 | 47.54
BLEX | 56.75 | 53.51 | 55.08 | 61.86
*** KoichiYasuoka/roberta-base-thai-spm-upos
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 90.69 | 88.49 | 89.58 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 90.69 | 88.49 | 89.58 |
UPOS | 74.54 | 72.73 | 73.62 | 82.19
XPOS | 90.69 | 88.49 | 89.58 | 100.00
UFeats | 87.39 | 85.28 | 86.32 | 96.37
AllTags | 71.51 | 69.78 | 70.63 | 78.85
Lemmas | 90.69 | 88.49 | 89.58 | 100.00
UAS | 69.88 | 68.19 | 69.03 | 77.06
LAS | 59.44 | 58.00 | 58.71 | 65.54
CLAS | 55.94 | 52.56 | 54.20 | 61.07
MLAS | 41.79 | 39.26 | 40.48 | 45.62
BLEX | 55.94 | 52.56 | 54.20 | 61.07
*** KoichiYasuoka/roberta-base-thai-syllable-upos
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 89.01 | 86.80 | 87.89 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 89.01 | 86.80 | 87.89 |
UPOS | 69.57 | 67.84 | 68.69 | 78.15
XPOS | 89.01 | 86.80 | 87.89 | 100.00
UFeats | 85.84 | 83.70 | 84.76 | 96.43
AllTags | 66.70 | 65.04 | 65.86 | 74.93
Lemmas | 89.01 | 86.80 | 87.89 | 100.00
UAS | 67.16 | 65.50 | 66.32 | 75.45
LAS | 57.23 | 55.81 | 56.51 | 64.30
CLAS | 53.45 | 50.29 | 51.82 | 59.71
MLAS | 37.08 | 34.89 | 35.95 | 41.42
BLEX | 53.45 | 50.29 | 51.82 | 59.71
*** KoichiYasuoka/roberta-base-thai-char-upos
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 88.44 | 86.65 | 87.53 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 88.44 | 86.65 | 87.53 |
UPOS | 70.49 | 69.06 | 69.77 | 79.71
XPOS | 88.44 | 86.65 | 87.53 | 100.00
UFeats | 85.29 | 83.56 | 84.42 | 96.44
AllTags | 67.66 | 66.29 | 66.97 | 76.51
Lemmas | 88.44 | 86.65 | 87.53 | 100.00
UAS | 66.52 | 65.17 | 65.84 | 75.21
LAS | 56.33 | 55.19 | 55.75 | 63.69
CLAS | 52.60 | 49.63 | 51.07 | 59.18
MLAS | 37.81 | 35.67 | 36.71 | 42.54
BLEX | 52.60 | 49.63 | 51.07 | 59.18
*** KoichiYasuoka/bert-base-thai-upos
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 79.12 | 69.92 | 74.23 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 79.12 | 69.92 | 74.23 |
UPOS | 51.69 | 45.69 | 48.50 | 65.34
XPOS | 79.12 | 69.92 | 74.23 | 100.00
UFeats | 76.13 | 67.28 | 71.43 | 96.22
AllTags | 49.44 | 43.69 | 46.39 | 62.49
Lemmas | 79.12 | 69.92 | 74.23 | 100.00
UAS | 44.54 | 39.36 | 41.79 | 56.29
LAS | 32.11 | 28.37 | 30.13 | 40.58
CLAS | 26.43 | 22.72 | 24.43 | 34.66
MLAS | 14.95 | 12.85 | 13.82 | 19.60
BLEX | 26.43 | 22.72 | 24.43 | 34.66
camembert-thai-base-uposのLAS/MLAS/BLEXが62.56/47.22/58.49で、まだまだ改善の余地がありそうだ。さて、どういう方法があるかな。