昨日の記事の続きだが、BERnaT-baseとUD_Basque-BDTをもとに、roberta-base-basque-ud-goeswithを試作してみた。Google Colaboratoryで動かしてみよう。
!pip install transformers deplacy
from transformers import pipeline
nlp=pipeline("universal-dependencies","KoichiYasuoka/roberta-base-basque-ud-goeswith",trust_remote_code=True,aggregation_strategy="simple")
doc=nlp("Gaur ama hil zen, gaur goizean ama hil nuen.")
import deplacy
deplacy.serve(doc,port=None)
「Gaur ama hil zen, gaur goizean ama hil nuen.」を係り受け解析してみたところ、私(安岡孝一)の手元では以下の結果が出力された。
# text = Gaur ama hil zen, gaur goizean ama hil nuen.
1 Gaur _ ADV _ _ 3 advmod _ _
2 ama _ NOUN _ Case=Abs|Definite=Def|Number=Sing 3 nsubj _ _
3 hil _ VERB _ Aspect=Perf|VerbForm=Part 0 root _ _
4 zen _ AUX _ Mood=Ind|Number[abs]=Sing|Person[abs]=3|VerbForm=Fin 3 aux _ SpaceAfter=No
5 , _ PUNCT _ _ 9 punct _ _
6 gaur _ ADV _ _ 9 advmod _ _
7 goizean _ NOUN _ Animacy=Inan|Case=Ine|Definite=Def|Number=Sing 9 obl _ _
8 ama _ NOUN _ Case=Abs|Definite=Def|Number=Sing 9 obj _ _
9 hil _ VERB _ Aspect=Perf|VerbForm=Part 3 conj _ _
10 nuen _ AUX _ Mood=Ind|Number[abs]=Sing|Number[erg]=Sing|Person[abs]=3|Person[erg]=1|VerbForm=Fin 9 aux _ SpaceAfter=No
11 . _ PUNCT _ _ 3 punct _ SpaceAfter=No
私個人としては悩ましいところもあるのだが、一応ちゃんと読めているようだ。roberta-small-basque-ud-goeswithとroberta-large-basque-ud-goeswithも試作したので、Google Colaboratoryで精度を比較してみよう。
!pip install transformers
models=[
"KoichiYasuoka/roberta-small-basque-ud-goeswith",
"KoichiYasuoka/roberta-base-basque-ud-goeswith",
"KoichiYasuoka/roberta-large-basque-ud-goeswith"
]
import os,sys,subprocess
url="https://github.com/UniversalDependencies/UD_Basque-BDT"
d=os.path.basename(url)
!test -d {d} || git clone --depth=1 {url}
os.system("for F in train dev test ; do sed 's/\u0092//' "+d+"/*-$F.conllu > $F.conllu ; done")
url="https://universaldependencies.org/conll18/conll18_ud_eval.py"
c=os.path.basename(url)
!test -f {c} || curl -LO {url}
for mdl in models:
from transformers import pipeline
nlp=pipeline("universal-dependencies",mdl,trust_remote_code=True,aggregation_strategy="simple")
for f in ["dev.conllu","test.conllu"]:
with open(f,"r",encoding="utf-8") as r:
s=[t[8:].strip() for t in r if t.startswith("# text =")]
with open("result-"+f,"w",encoding="utf-8") as w:
for t in s:
w.write(nlp(t))
os.system(f"mkdir -p result/{mdl}")
with open(f"result/{mdl}/result.txt","w",encoding="utf-8") as w:
for f in ["dev.conllu","test.conllu"]:
p=subprocess.run([sys.executable,c,"-v",f,"result-"+f],encoding="utf-8",stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
print(f"\n*** {mdl} ({f})",p.stdout,sep="\n",file=w)
!( cd result && cat `find {" ".join(models)} -name result.txt` )
eu_bdt-ud-dev.conlluとeu_bdt-ud-test.conlluの両方で評価したところ、私の手元では以下の結果が出力された。
*** KoichiYasuoka/roberta-small-basque-ud-goeswith (dev.conllu)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.45 | 98.96 | 99.20 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.45 | 98.96 | 99.20 |
UPOS | 93.49 | 93.03 | 93.26 | 94.01
XPOS | 99.45 | 98.96 | 99.20 | 100.00
UFeats | 84.19 | 83.78 | 83.98 | 84.66
AllTags | 82.15 | 81.75 | 81.95 | 82.61
Lemmas | 0.12 | 0.12 | 0.12 | 0.12
UAS | 87.69 | 87.26 | 87.47 | 88.18
LAS | 83.20 | 82.79 | 83.00 | 83.66
CLAS | 81.45 | 81.20 | 81.33 | 81.24
MLAS | 64.38 | 64.18 | 64.28 | 64.21
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/roberta-small-basque-ud-goeswith (test.conllu)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.47 | 98.98 | 99.23 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.47 | 98.98 | 99.23 |
UPOS | 93.52 | 93.06 | 93.29 | 94.02
XPOS | 99.47 | 98.98 | 99.23 | 100.00
UFeats | 84.46 | 84.04 | 84.25 | 84.91
AllTags | 82.42 | 82.02 | 82.22 | 82.86
Lemmas | 0.10 | 0.10 | 0.10 | 0.10
UAS | 88.54 | 88.11 | 88.32 | 89.01
LAS | 84.04 | 83.63 | 83.83 | 84.49
CLAS | 82.33 | 81.98 | 82.16 | 82.03
MLAS | 65.20 | 64.93 | 65.07 | 64.97
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/roberta-base-basque-ud-goeswith (dev.conllu)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.38 | 98.95 | 99.16 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.38 | 98.95 | 99.16 |
UPOS | 95.73 | 95.31 | 95.52 | 96.33
XPOS | 99.38 | 98.95 | 99.16 | 100.00
UFeats | 90.65 | 90.26 | 90.45 | 91.22
AllTags | 89.08 | 88.69 | 88.88 | 89.64
Lemmas | 0.12 | 0.12 | 0.12 | 0.12
UAS | 86.97 | 86.59 | 86.78 | 87.52
LAS | 83.51 | 83.15 | 83.33 | 84.03
CLAS | 84.01 | 84.05 | 84.03 | 84.11
MLAS | 73.61 | 73.65 | 73.63 | 73.70
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/roberta-base-basque-ud-goeswith (test.conllu)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 99.25 | 98.92 | 99.08 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 99.25 | 98.92 | 99.08 |
UPOS | 95.37 | 95.06 | 95.21 | 96.10
XPOS | 99.25 | 98.92 | 99.08 | 100.00
UFeats | 90.33 | 90.03 | 90.18 | 91.01
AllTags | 88.58 | 88.28 | 88.43 | 89.25
Lemmas | 0.10 | 0.10 | 0.10 | 0.10
UAS | 87.70 | 87.41 | 87.55 | 88.37
LAS | 84.56 | 84.28 | 84.42 | 85.21
CLAS | 85.16 | 85.26 | 85.21 | 85.40
MLAS | 74.16 | 74.25 | 74.20 | 74.37
BLEX | 0.01 | 0.01 | 0.01 | 0.01
*** KoichiYasuoka/roberta-large-basque-ud-goeswith (dev.conllu)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 97.87 | 96.48 | 97.17 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 97.87 | 96.48 | 97.17 |
UPOS | 89.88 | 88.60 | 89.23 | 91.83
XPOS | 97.87 | 96.48 | 97.17 | 100.00
UFeats | 85.31 | 84.10 | 84.70 | 87.16
AllTags | 83.65 | 82.46 | 83.05 | 85.46
Lemmas | 0.12 | 0.12 | 0.12 | 0.12
UAS | 82.60 | 81.42 | 82.00 | 84.39
LAS | 76.19 | 75.10 | 75.64 | 77.84
CLAS | 80.62 | 81.11 | 80.86 | 82.81
MLAS | 65.37 | 65.76 | 65.56 | 67.14
BLEX | 0.00 | 0.00 | 0.00 | 0.00
*** KoichiYasuoka/roberta-large-basque-ud-goeswith (test.conllu)
Metric | Precision | Recall | F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens | 97.92 | 96.79 | 97.35 |
Sentences | 100.00 | 100.00 | 100.00 |
Words | 97.92 | 96.79 | 97.35 |
UPOS | 89.85 | 88.82 | 89.33 | 91.76
XPOS | 97.92 | 96.79 | 97.35 | 100.00
UFeats | 85.45 | 84.47 | 84.96 | 87.27
AllTags | 83.64 | 82.67 | 83.15 | 85.42
Lemmas | 0.10 | 0.10 | 0.10 | 0.11
UAS | 83.43 | 82.47 | 82.95 | 85.21
LAS | 77.32 | 76.43 | 76.87 | 78.97
CLAS | 81.37 | 82.09 | 81.73 | 83.67
MLAS | 66.14 | 66.73 | 66.43 | 68.01
BLEX | 0.01 | 0.01 | 0.01 | 0.01
UPOS/LAS/MLASを表にしてみよう。
| eu_bdt-ud-dev.conllu | eu_bdt-ud-test.conllu | |
|---|---|---|
| roberta-small-basque-ud-goeswith | 93.26/83.00/64.28 | 93.29/83.83/65.07 |
| roberta-base-basque-ud-goeswith | 95.52/83.33/73.63 | 95.21/84.42/74.20 |
| roberta-large-basque-ud-goeswith | 89.23/75.64/65.56 | 89.33/76.87/66.43 |
roberta-base-basque-ud-goeswithが健闘しているものの、他の2モデルはイマイチふるわない。ただ、バスク語Universal Dependenciesは、能格言語としての特徴をあまり押し出していないので、係り受け解析モデルをどう作っていくかは、もう少し考える必要がありそうだ。