始めに
この記事はR AdventCalendar 20日目の記事です。
言語処理以外の記事を書こうと思ったのですが、諸々の進捗が許してくれませんでした。ということで、今回も言語処理系のお話です。
本日は{quanteda}というテキスト処理・解析のためのRパッケージについてご紹介します。
{quanteda}とは
A fast, flexible toolset for for the management, processing, and quantitative analysis of textual data in R.
テキストファイルからのコーパス作成からトークン化・ステミング、N-gramや類似度に可読性指標の計算など、言語処理タスクを手早く手軽にしやすくするためのRパッケージです（ただし、英語文書の解析がメインです）。
下記がパッケージのリンクです。上がCRANで下がGitHubです。
quanteda: Quantitative Analysis of Textual Data
kbenoit/quanteda
テキスト処理には{stringr}と{stringi}を用いており、{data.table}による大規模な文書のインデックス化と{Matrix}による疎行列化がされています。同じようなテキスト処理用フレームワークに{tm}もありますが、{tm}で作成したオブジェクトも一部利用できます。
{quanteda}で何ができるか？
-
コーパス管理
- テキストファイル処理（個別ファイルからディレクトリ単位でのテキストの読み込みや、ヘルパー関数の定義）
- ある言語単位（文や文書など）での集計や抽出、リサンプリング（ノンパラメトリックブートスラッピング）
- KeyWords In Context (KWIC)
-
言語処理ツール
- タイプ数やN-gramなどのテキストの素性量抽出や、ユーザー定義の辞書やシソーラス作成
- ステミングやランダム選択、文書頻度、単語頻度などによる素性の絞り込み
- 英単語用前処理
- 可読性指標や語彙の多様性指標、連語分析、類似度算出
- TFやTF-IDF以外にもコレスポンデンス分析やWordfish modelによる素性の重み付け
- （トピックモデルやNaive Bayesやk-nearest neighbourは現在は未実装）
Rで実行する
定義されている定数や関数を、一部ですが試してみます。
quanteda-misc.R
library(dplyr)
library(quanteda)
# The stopword list are SMART English stopwords from the SMART information retrieval system
> quanteda::stopwords(kind = "english") %>%
+ head(n = 5)
[1] "i" "me" "my" "myself" "we"
# 英語の音節
# ex・am・ple
> quanteda::syllables(x = "example")
[1] 3
# sta・tis・ti・cal
> quanteda::syllables(x = "statistical")
[1] 4
# {SnowballC}によるポーターステミング
> quanteda::wordstem(x = c("win", "winning", "wins", "won", "winner"), language = "porter")
[1] "win" "win" "win" "won" "winner"
# N-gram
# 日本語もわかち書きしてもおかしい結果を出すケースがある。
> quanteda::ngrams(
+ text = c("すもも も もも も もも の うち", "吾輩 は 猫 で ある 。", "Why are you using SJIS ?"),
+ n = 2, concatenator = "-"
+ )
[[1]]
[1] "すもも-も" "も-も" "も-も" "も-も" "も-も" "も-も" "も-の" "の-うち"
[[2]]
[1] "吾輩-は" "は-猫" "猫-で" "で-ある" "ある-。"
[[3]]
[1] "Why-are" "are-you" "you-using" "using-SJIS" "SJIS-?"
- コーパス管理
quanteda-corpus.R
library(dplyr)
library(tm)
library(quanteda)
# {tm}のデータセットcrudeを利用
data(crude)
crude_coupus <- quanteda::corpus(x = crude, encTo = "UTF-8")
> class(x = crude_coupus)
[1] "corpus" "list"
> summary(object = crude_coupus)
Corpus consisting of 20 documents.
Text Types Tokens Sentences
127 58 92 12
144 227 443 52
191 43 55 9
194 52 69 10
211 64 93 12
236 237 458 57
237 236 431 60
242 110 154 18
246 181 324 44
248 187 344 41
273 192 373 44
349 67 92 12
352 70 105 12
353 72 98 12
368 72 109 14
489 94 148 17
502 121 198 22
543 51 83 10
704 142 281 33
708 38 53 8
Source: Converted from tm VCorpus 'crude'.
Created: Sat Dec 19 15:52:18 2015.
Notes: .
# コーパスの一部を取り出す
> stringr::str_split(string = quanteda::texts(x = crude_coupus)[1], pattern = "\n")
[[1]]
[1] "Diamond Shamrock Corp said that"
[2] "effective today it had cut its contract prices for crude oil by"
[3] "1.50 dlrs a barrel."
[4] " The reduction brings its posted price for West Texas"
[5] "Intermediate to 16.00 dlrs a barrel, the copany said."
[6] " \"The price reduction today was made in the light of falling"
[7] "oil product prices and a weak crude oil market,\" a company"
[8] "spokeswoman said."
[9] " Diamond is the latest in a line of U.S. oil companies that"
[10] "have cut its contract, or posted, prices over the last two days"
[11] "citing weak oil markets."
[12] " Reuter"
# 様々な処理ができるトークン化
> tokenized_crude_coupus <- quanteda::tokenize(
+ x = crude_coupus, what = "sentence",
+ removeNumbers = FALSE, removePunct = FALSE, removeSeparators = TRUE, removeTwitter = FALSE,
+ ngrams = 1,
+ verbose = TRUE
+ )
Starting tokenization...
...preserving Twitter characters (#, @)...total elapsed: 0 seconds.
...tokenizing texts
...separating into sentences....total elapsed: 0.008 seconds.
...replacing Twitter characters (#, @)...total elapsed: 0 seconds.
...replacing names...total elapsed: 0.001 seconds.
Finished tokenizing and cleaning 20 texts.
> class(x = tokenized_crude_coupus)
[1] "tokenizedTexts" "list"
# 各文書の可読性指標
> quanteda::readability(x = crude_coupus, measure = "all") %>%
+ head(n = 2)
ARI ARI.NRI ARI.simple Bormuth Bormuth.GP Coleman Coleman.C2 Coleman.Liau
127 3.495942 2.536232 47.97101 -1.090237 2955495 41.47391 53.22391 59.82560
144 5.018712 4.274058 50.91878 -1.283181 5139826 35.80508 46.19447 53.42044
Coleman.Liau.grade Coleman.Liau.short Dale.Chall Dale.Chall.old Dale.Chall.PSK Danielson.Bryan
127 6.671495 6.671304 25.66652 -30.60794 -30.94298 4.752439
144 8.426537 8.426546 22.30909 -33.48056 -33.80707 5.066302
Danielson.Bryan.2 Dickes.Steiwer DRP ELF Farr.Jenkins.Paterson Flesch Flesch.PSK
127 84.80234 199.5486 209.0237 2.583333 -40.28935 80.42942 4.773458
144 82.90171 211.3242 228.3181 3.384615 -41.08444 65.27241 5.608429
Flesch.Kincaid FOG FOG.PSK FOG.NRI FORCAST FORCAST.RGL Fucks Linsear.Write LIW
127 3.945652 6.544928 2.918473 4.058333 10.70652 10.20717 34.33333 -0.41666667 23.97101
144 6.271552 9.096180 3.691583 33.514038 11.36569 10.93226 40.13462 -0.01923077 31.54406
nWS nWS.2 nWS.3 nWS.4 RIX SMOG SMOG.C SMOG.simple SMOG.de Spache
127 4.010888 4.434446 2.922622 2.729354 1.250000 7.793538 7.872500 7.472136 2.472136 5.686667
144 5.781312 6.126124 4.722261 4.472010 1.961538 9.417115 9.300595 9.028777 4.028777 5.336328
Spache.old Strain Traenkle.Bailer Traenkle.Bailer.2 Wheeler.Smith
127 6.220000 3.225000 -246.3363 -232.0354 25.83333
144 5.864591 4.015385 -277.8625 -258.7959 33.84615
# 連語とそのスコア
> quanteda::collocations(x = crude_coupus, method = "all", size = 2)
word1 word2 word3 count G2 X2 pmi dice
1: mln bpd 14 1.213317e+02 1.270675e+03 4.5168446312 0.58333333
2: a barrel 14 1.030286e+02 5.989800e+02 3.7767238174 0.30434783
3: dlrs a 13 8.190325e+01 4.497635e+02 3.5759259844 0.29545455
4: billion riyals 6 7.976580e+01 2.700747e+03 6.1100805691 0.85714286
5: crude oil 13 7.626258e+01 3.749103e+02 3.4020303680 0.26530612
---
2589: of a 2 1.117296e-02 1.143404e-02 0.0738518082 0.02409639
2590: in its 1 2.194860e-03 2.220715e-03 0.0462953604 0.01587302
2591: said a 1 2.517813e-05 1.800434e-05 0.0041718876 0.01652893
2592: was to 1 1.370888e-05 6.505725e-06 -0.0024948038 0.01273885
2593: and the 4 1.070793e-05 3.510900e-06 0.0009024332 0.03030303
# KWIC (KeyWord In Context)
> quanteda::kwic(x = crude_coupus, word = "will", window = 3)
preword word postword
[144, 266] "They will not meet now
[144, 327] next two months will be critical for\nOPEC's
[144, 355] eight weeks\nsince buyers will come back into
[191, 9] the\ncontract price it will pay for crude
[194, 10] contract price it will pay for all
[236, 163] the ability, it will do so," the
[236, 450] such pressure. It will continue through March
[242, 31] over whether OPEC will succeed in halting\nthe
[248, 56] Accord and it will never sell its
[273, 263] a weak market will\ncome this month, when
[349, 8] six Gulf\nArab states will meet in Bahrain
[352, 56] accord and it will never sell its
[704, 19] the energy\ncomplex that will increase the use
[704, 35] April one, NYMEX will allow oil traders
[704, 69] "This will change the way
[704, 92] Foreign traders will be able to
[704, 121] The expanded program "will serve the industry
[704, 257] of the EFP\nprovision will add to globalization
- 言語処理ツール
quanteda-dfm.R
library(dplyr)
library(quanteda)
# ストップワードと"will"を除去し、{quanteda}の関数が適用しやすいdfmオブジェクトへ変換
> crude_coupus_ignore_stopwords <- quanteda::dfm(
+ x = crude_coupus,
+ ignoredFeatures = c("will", quanteda::stopwords(kind = "english")),
+ stem = TRUE, matrixType = "sparse"
+ )
Creating a dfm from a corpus ...
... lowercasing
... tokenizing
... indexing 20 documents
... indexing 1,052 feature types
... removed 89 features, from 175 supplied feature types
... stemming features (English), trimmed 169 feature variants
... created a 20 x 794 sparse dfm
... complete.
Elapsed time: 0.056 seconds.
> class(x = crude_coupus_ignore_stopwords)
[1] "dfmSparse"
attr(,"package")
[1] "quanta"
# summary(object = crude_coups)のときよりも減っている
> quanteda::ntype(x = crude_coupus_ignore_stopwords)
127 144 191 194 211 236 237 242 246 248 273 349 352 353 368 489 502 543 704 708
36 158 31 33 37 157 179 70 121 128 134 49 49 47 48 60 78 33 101 24
> quanteda::ntoken(x = crude_coupus_ignore_stopwords)
127 144 191 194 211 236 237 242 246 248 273 349 352 353 368 489 502 543 704 708
58 262 38 44 54 263 268 86 188 209 242 65 68 60 66 86 113 54 172 35
# dfmオブジェクトと型変換
# dfmSparse
> str(object = crude_coupus_ignore_stopwords)
Formal class 'dfmSparse' [package "quanteda"] with 9 slots
..@ settings :List of 1
.. ..$ : NULL
..@ weighting: chr "frequency"
..@ smooth : num 0
..@ Dim : int [1:2] 20 794
..@ Dimnames :List of 2
.. ..$ docs : chr [1:20] "127" "144" "191" "194" ...
.. ..$ features: chr [1:794] "diamond" "shamrock" "corp" "said" ...
..@ i : int [1:15880] 0 1 2 3 4 5 6 7 8 9 ...
..@ p : int [1:795] 0 20 40 60 80 100 120 140 160 180 ...
..@ x : num [1:15880] 2 0 0 0 0 0 0 0 0 0 ...
..@ factors : list()
# austin's wfm format
> str(object = quanteda::as.wfm(x = crude_coupus_ignore_stopwords))
wfm [1:20, 1:794] 2 0 0 0 0 0 0 0 0 0 ...
- attr(*, "dimnames")=List of 2
..$ docs : chr [1:20] "127" "144" "191" "194" ...
..$ words: chr [1:794] "diamond" "shamrock" "corp" "said" ...
- attr(*, "class")= chr [1:2] "wfm" "matrix"
# tm's DocumentTermMatrix format
> str(object = quanteda::as.DocumentTermMatrix(x = crude_coupus_ignore_stopwords))
List of 6
$ i : int [1:15880] 1 2 3 4 5 6 7 8 9 10 ...
$ j : int [1:15880] 1 1 1 1 1 1 1 1 1 1 ...
$ v : num [1:15880] 2 0 0 0 0 0 0 0 0 0 ...
$ nrow : int 20
$ ncol : int 794
$ dimnames:List of 2
..$ Docs : chr [1:20] "127" "144" "191" "194" ...
..$ Terms: chr [1:794] "diamond" "shamrock" "corp" "said" ...
- attr(*, "class")= chr [1:2] "DocumentTermMatrix" "simple_triplet_matrix"
- attr(*, "weighting")= chr [1:2] "term frequency" "tf"
# stm package format
> str(object = quanteda::convert(crude_coupus_ignore_stopwords, to = "stm"))
List of 3
$ documents:List of 20
..$ 127: int [1:2, 1:36] 1 2 2 1 3 1 4 3 5 1 ...
..$ 144: int [1:2, 1:158] 4 11 7 1 9 6 11 12 24 6 ...
..$ 191: int [1:2, 1:31] 4 1 5 1 6 1 8 1 9 2 ...
..$ 194: int [1:2, 1:33] 4 1 5 1 6 1 8 1 9 2 ...
..$ 211: int [1:2, 1:37] 4 3 11 1 12 2 21 1 36 1 ...
..$ 236: int [1:2, 1:157] 4 10 6 1 9 8 10 2 11 7 ...
..$ 237: int [1:2, 1:179] 4 1 8 1 9 1 11 3 12 1 ...
..$ 242: int [1:2, 1:70] 4 3 9 2 11 3 26 2 36 1 ...
..$ 246: int [1:2, 1:121] 4 5 6 1 9 2 11 5 13 1 ...
..$ 248: int [1:2, 1:128] 4 7 9 10 11 9 12 4 13 3 ...
..$ 273: int [1:2, 1:134] 3 3 4 8 7 1 9 5 10 5 ...
..$ 349: int [1:2, 1:49] 4 1 6 1 9 1 10 2 11 4 ...
..$ 352: int [1:2, 1:49] 4 2 7 1 9 5 11 5 13 1 ...
..$ 353: int [1:2, 1:47] 4 1 9 2 10 2 11 4 13 1 ...
..$ 368: int [1:2, 1:48] 4 3 6 1 11 3 30 2 36 1 ...
..$ 489: int [1:2, 1:60] 4 2 9 3 11 4 12 1 13 3 ...
..$ 502: int [1:2, 1:78] 4 2 9 3 11 5 12 1 13 3 ...
..$ 543: int [1:2, 1:33] 3 1 4 4 5 1 7 1 9 3 ...
..$ 704: int [1:2, 1:101] 4 4 5 2 8 1 9 3 11 3 ...
..$ 708: int [1:2, 1:24] 4 1 10 1 11 1 13 2 24 1 ...
$ vocab : chr [1:794] "13-member" "13-nation" "200-foot" "20s" ...
$ meta : NULL
# topicmodels package format
> str(object = quanteda::quantedaformat2dtm(x = crude_coupus_ignore_stopwords))
List of 6
$ i : int [1:15880] 1 1 1 1 1 1 1 1 1 1 ...
$ j : int [1:15880] 1 2 3 4 5 6 7 8 9 10 ...
$ v : int [1:15880] 2 1 1 3 1 2 2 2 5 2 ...
$ nrow : int 20
$ ncol : int 794
$ dimnames:List of 2
..$ Docs : chr [1:20] "127" "144" "191" "194" ...
..$ Terms: chr [1:794] "diamond" "shamrock" "corp" "said" ...
- attr(*, "class")= chr [1:2] "DocumentTermMatrix" "simple_triplet_matrix"
- attr(*, "weighting")= chr [1:2] "term frequency" "tf"
# lda package format
> str(object = quanteda::dfm2ldaformat(x = crude_coupus_ignore_stopwords))
List of 2
$ documents:List of 20
..$ 127: int [1:2, 1:794] 0 2 1 1 2 1 3 3 4 1 ...
..$ 144: int [1:2, 1:794] 0 0 1 0 2 0 3 11 4 0 ...
..$ 191: int [1:2, 1:794] 0 0 1 0 2 0 3 1 4 1 ...
..$ 194: int [1:2, 1:794] 0 0 1 0 2 0 3 1 4 1 ...
..$ 211: int [1:2, 1:794] 0 0 1 0 2 0 3 3 4 0 ...
..$ 236: int [1:2, 1:794] 0 0 1 0 2 0 3 10 4 0 ...
..$ 237: int [1:2, 1:794] 0 0 1 0 2 0 3 1 4 0 ...
..$ 242: int [1:2, 1:794] 0 0 1 0 2 0 3 3 4 0 ...
..$ 246: int [1:2, 1:794] 0 0 1 0 2 0 3 5 4 0 ...
..$ 248: int [1:2, 1:794] 0 0 1 0 2 0 3 7 4 0 ...
..$ 273: int [1:2, 1:794] 0 0 1 0 2 3 3 8 4 0 ...
..$ 349: int [1:2, 1:794] 0 0 1 0 2 0 3 1 4 0 ...
..$ 352: int [1:2, 1:794] 0 0 1 0 2 0 3 2 4 0 ...
..$ 353: int [1:2, 1:794] 0 0 1 0 2 0 3 1 4 0 ...
..$ 368: int [1:2, 1:794] 0 0 1 0 2 0 3 3 4 0 ...
..$ 489: int [1:2, 1:794] 0 0 1 0 2 0 3 2 4 0 ...
..$ 502: int [1:2, 1:794] 0 0 1 0 2 0 3 2 4 0 ...
..$ 543: int [1:2, 1:794] 0 0 1 0 2 1 3 4 4 1 ...
..$ 704: int [1:2, 1:794] 0 0 1 0 2 0 3 4 4 2 ...
..$ 708: int [1:2, 1:794] 0 0 1 0 2 0 3 1 4 0 ...
$ vocab : chr [1:794] "diamond" "shamrock" "corp" "said" ...
# サンプリング
> set.seed(seed = 71)
# corpusオブジェクト
> summary(quanteda::sample(x = crude_coupus, size = 5))
Corpus consisting of 5 documents.
Text Types Tokens Sentences
237 236 431 60
273 192 373 44
236 237 458 57
194 52 69 10
543 51 83 10
Source: Converted from tm VCorpus 'crude'.
Created: Sat Dec 19 15:52:18 2015.
Notes: .
# dfmオブジェクトでfeature単位
> summary(quanteda::sample(x = crude_coupus_ignore_stopwords, size = 5, what = "features"))
20 x 5 sparse Matrix of class "dfmSparse", with 100 entries
i j x
1 1 1 0
2 2 1 0
3 3 1 0
4 4 1 0
5 5 1 0
6 6 1 0
7 7 1 0
8 8 1 0
9 9 1 0
10 10 1 0
11 11 1 0
12 12 1 0
13 13 1 0
14 14 1 0
15 15 1 0
16 16 1 0
17 17 1 0
18 18 1 0
19 19 1 1
20 20 1 0
21 1 2 0
22 2 2 0
23 3 2 0
24 4 2 0
25 5 2 0
26 6 2 0
27 7 2 0
28 8 2 0
29 9 2 1
30 10 2 0
31 11 2 0
32 12 2 0
33 13 2 0
34 14 2 0
35 15 2 0
36 16 2 0
37 17 2 0
38 18 2 0
39 19 2 0
40 20 2 0
41 1 3 0
42 2 3 0
43 3 3 0
44 4 3 0
45 5 3 0
46 6 3 0
47 7 3 0
48 8 3 0
49 9 3 0
50 10 3 0
51 11 3 0
52 12 3 0
53 13 3 0
54 14 3 0
55 15 3 0
56 16 3 1
57 17 3 1
58 18 3 0
59 19 3 0
60 20 3 0
61 1 4 0
62 2 4 0
63 3 4 0
64 4 4 0
65 5 4 0
66 6 4 1
67 7 4 0
68 8 4 0
69 9 4 0
70 10 4 0
71 11 4 0
72 12 4 0
73 13 4 0
74 14 4 0
75 15 4 0
76 16 4 0
77 17 4 0
78 18 4 0
79 19 4 0
80 20 4 0
81 1 5 0
82 2 5 0
83 3 5 0
84 4 5 0
85 5 5 0
86 6 5 0
87 7 5 3
88 8 5 0
89 9 5 0
90 10 5 0
91 11 5 2
92 12 5 0
93 13 5 0
94 14 5 0
95 15 5 0
96 16 5 0
97 17 5 0
98 18 5 0
99 19 5 0
100 20 5 0
# 頻度ベースで上位の素性を抽出（10回以上出現して、2文書以上に出現するものに限定）
> quanteda::topfeatures(
+ x = quanteda::trim(x = crude_coupus_ignore_stopwords, minCount = 10, minDoc = 2)
+ )
Features occurring less than 10 times: 758
Features occurring in fewer than 2 documents: 489
oil said price opec mln market barrel last dlrs bpd
85 73 63 47 31 30 26 24 23 23
# 10回以上出現する素性をサンプリング
> quanteda::trim(x = crude_coupus_ignore_stopwords, minCount = 10, nsample = 5)
Features occurring less than 10 times: 758
Retaining a random sample of 5 words
Document-feature matrix of: 20 documents, 5 features.
20 x 5 sparse Matrix of class "dfmSparse"
features
docs dlrs offici accord s sheikh
127 2 0 0 0 0
144 0 0 0 0 0
191 1 0 0 0 0
194 2 0 0 0 0
211 2 0 0 0 0
236 2 5 0 5 3
237 1 1 0 6 0
242 0 1 0 0 0
246 0 0 0 0 5
248 4 1 5 0 2
273 2 4 1 0 0
349 0 3 0 0 0
352 0 1 2 0 0
353 0 0 0 0 1
368 0 0 0 0 0
489 1 0 0 0 0
502 1 0 0 0 0
543 5 0 0 0 0
704 0 1 4 0 0
708 0 0 0 0 0
# 頻度ベースで上位10個の素性を抽出
> quanteda::topfeatures(x = crude_coupus_ignore_stopwords, n = 10, decreasing = TRUE)
oil said price opec mln market barrel last dlrs bpd
85 73 63 47 31 30 26 24 23 23
> plot(crude_coupus_ignore_stopwords, min.freq = 10, random.order = FALSE)
# 素性の重み付けを変える
> log_tf <- quanteda::weight(x = crude_coupus_ignore_stopwords, type = "logFreq")
> wfidf <- quanteda::weight(x = log_tf, type = "tfidf", normalize = FALSE)
# "weighting"の属性値が変わっている
> str(object = wfidf)
dfm [1:20, 1:794] 3.29 0 0 0 0 ...
- attr(*, "dimnames")=List of 2
..$ docs : chr [1:20] "127" "144" "191" "194" ...
..$ features: chr [1:794] "diamond" "shamrock" "corp" "said" ...
- attr(*, "class")= chr [1:2] "dfm" "matrix"
- attr(*, "weighting")= chr "tfidf"
# 重みと順番が変わっている
> quanteda::topfeatures(x = log_tf, n = 10, decreasing = TRUE)
oil said price opec reuter barrel mln market last dlrs
30.84361 27.17000 24.34084 15.40452 14.26841 13.51072 12.57072 12.04984 11.71896 11.66685
> quanteda::topfeatures(x = wfidf, n = 10, decreasing = TRUE)
bpd opec mln market saudi kuwait s govern accord crude
11.205873 10.677598 10.037813 9.621890 9.519591 8.690813 8.606302 8.606302 8.357742 8.354686
# 定義されている重み付けの全種類を試す
> sapply(
+ X = mapply(
+ x = rep(x = list(crude_coupus_ignore_stopwords), each = 5),
+ type = c("frequency", "relFreq", "relMaxFreq", "logFreq", "tfidf"),
+ FUN = quanteda::weight,
+ smooth = 0
+ ),
+ FUN = quanteda::topfeatures,
+ n = 10, decreasing = TRUE
+ ) %>%
+ t
oil said price opec mln market barrel last dlrs
[1,] 85.0000000 73.0000000 63.0000000 47.0000000 31.0000000 30.0000000 26.0000000 24.0000000 23.0000000
[2,] 0.8284160 0.6418656 0.6004227 0.3132729 0.3015126 0.2992712 0.2950993 0.2611242 0.2413641
[3,] 14.3948413 11.3892857 10.3662698 5.7662698 4.9095238 4.8055556 4.6305556 4.6083333 4.2900794
[4,] 30.8436127 27.1700007 24.3408417 15.4045172 14.2684087 13.5107230 12.5707158 12.0498398 11.7189635
[5,] 0.3027746 0.3012118 0.2501508 0.2173108 0.2089926 0.2064206 0.1968457 0.1927311 0.1922584
bpd
[1,] 23.0000000
[2,] 0.2178314
[3,] 3.8777778
[4,] 11.6668475
[5,] 0.1892054
# 類似度を計算
> quanteda::similarity(
+ x = crude_coupus_ignore_stopwords,
+ selection = c("oil", "price"), n = 10, margin = "features",
+ method = "cosine"
+ )
$price
oil said reuter compani cut last barrel market effect bring
0.8969 0.7757 0.7409 0.7337 0.7294 0.7152 0.7045 0.6904 0.6751 0.6717
$oil
price reuter said barrel market last opec crude cut today
0.8969 0.8531 0.8507 0.7137 0.6845 0.6765 0.6443 0.6283 0.6163 0.6094
# 語彙の多様性
# 文書単位で計算
> mapply(
+ x = rep(x = list(crude_coupus_ignore_stopwords), each = 7),
+ type = c("TTR", "C", "R", "CTTR", "U", "S", "Maas"),
+ FUN = quanteda::lexdiv,
+ log.base = 10
+ )
[,1] [,2] [,3] [,4] [,5] [,6] [,7]
127 0.6206897 0.6206897 0.6206897 0.6206897 0.6206897 0.6206897 0.6206897
144 0.6030534 0.6030534 0.6030534 0.6030534 0.6030534 0.6030534 0.6030534
191 0.8157895 0.8157895 0.8157895 0.8157895 0.8157895 0.8157895 0.8157895
194 0.7500000 0.7500000 0.7500000 0.7500000 0.7500000 0.7500000 0.7500000
211 0.6851852 0.6851852 0.6851852 0.6851852 0.6851852 0.6851852 0.6851852
236 0.5969582 0.5969582 0.5969582 0.5969582 0.5969582 0.5969582 0.5969582
237 0.6679104 0.6679104 0.6679104 0.6679104 0.6679104 0.6679104 0.6679104
242 0.8139535 0.8139535 0.8139535 0.8139535 0.8139535 0.8139535 0.8139535
246 0.6436170 0.6436170 0.6436170 0.6436170 0.6436170 0.6436170 0.6436170
248 0.6124402 0.6124402 0.6124402 0.6124402 0.6124402 0.6124402 0.6124402
273 0.5537190 0.5537190 0.5537190 0.5537190 0.5537190 0.5537190 0.5537190
349 0.7538462 0.7538462 0.7538462 0.7538462 0.7538462 0.7538462 0.7538462
352 0.7205882 0.7205882 0.7205882 0.7205882 0.7205882 0.7205882 0.7205882
353 0.7833333 0.7833333 0.7833333 0.7833333 0.7833333 0.7833333 0.7833333
368 0.7272727 0.7272727 0.7272727 0.7272727 0.7272727 0.7272727 0.7272727
489 0.6976744 0.6976744 0.6976744 0.6976744 0.6976744 0.6976744 0.6976744
502 0.6902655 0.6902655 0.6902655 0.6902655 0.6902655 0.6902655 0.6902655
543 0.6111111 0.6111111 0.6111111 0.6111111 0.6111111 0.6111111 0.6111111
704 0.5872093 0.5872093 0.5872093 0.5872093 0.5872093 0.5872093 0.5872093
708 0.6857143 0.6857143 0.6857143 0.6857143 0.6857143 0.6857143 0.6857143
# quanteda::textmodel()のexample
> ie2010Corpus_dfm <- quanteda::dfm(x = ie2010Corpus, verbose = FALSE)
> ref_scores <- c(rep(x = NA, each = 4), -1, 1, rep(x = NA, each = 8))
# Wordfish model
> ws <- quanteda::textmodel(
+ x = ie2010Corpus_dfm, y = ref_scores, model = "wordscores", smooth = 1
+ )
# Wordfish model: scale = "logit"
> bs <- quanteda::textmodel(
+ x = ie2010Corpus_dfm, y = ref_scores, model = "wordscores", smooth = 1, scale = "logit"
+ )
> plot(
+ x = ws@Sw, y = bs@Sw, xlim = c(-1, 1),
+ xlab = "Linear word score", ylab = "Logit word score"
+ )
> wordfish <- quanteda::textmodel(
+ x = ie2010Corpus_dfm, y = NULL, model = "wordfish"
+ ) %>%
+ print
Fitted wordfish model:
Call:
textmodel_wordfish(data = x)
Estimated document positions:
Documents theta SE lower upper
1 2010_BUDGET_01_Brian_Lenihan_FF -1.77842508 NA NA NA
2 2010_BUDGET_02_Richard_Bruton_FG 0.58436869 NA NA NA
3 2010_BUDGET_03_Joan_Burton_LAB 1.14761695 NA NA NA
4 2010_BUDGET_04_Arthur_Morgan_SF 0.09400958 NA NA NA
5 2010_BUDGET_05_Brian_Cowen_FF -1.79211539 NA NA NA
6 2010_BUDGET_06_Enda_Kenny_FG 0.78894787 NA NA NA
7 2010_BUDGET_07_Kieran_ODonnell_FG 0.49306437 NA NA NA
8 2010_BUDGET_08_Eamon_Gilmore_LAB 0.58812988 NA NA NA
9 2010_BUDGET_09_Michael_Higgins_LAB 0.97901464 NA NA NA
10 2010_BUDGET_10_Ruairi_Quinn_LAB 0.92084329 NA NA NA
11 2010_BUDGET_11_John_Gormley_Green -1.12261547 NA NA NA
12 2010_BUDGET_12_Eamon_Ryan_Green -0.21004677 NA NA NA
13 2010_BUDGET_13_Ciaran_Cuffe_Green -0.79133534 NA NA NA
14 2010_BUDGET_14_Caoimhghin_OCaolain_SF 0.09854280 NA NA NA
Estimated feature scores: showing first 30 beta-hats for features
when i presented the supplementary budget
0.13824923 -0.33876765 -0.35600050 -0.21346691 -1.07842554 -0.05457924
to this house last april said
-0.33005556 -0.26757662 -0.16145904 -0.25783983 0.15467183 0.82440376
we could work our way through
-0.44459300 0.57604718 -0.54806584 -0.71558503 -0.29671938 -0.64350927
period of severe economic distress today
-0.52533975 -0.29754960 -1.24430279 -0.44508315 -1.78752195 -0.11651167
can report that notwithstanding difficulties past
-0.33001557 -0.65601898 -0.04671622 -1.78752195 -1.18563671 -0.50345524
最後に
{quanteda}というテキスト処理・解析のためのRパッケージについて、一部の関数を実際に動かしてみました。今回紹介した以外にも様々なテキスト分析の関数が定義されており、手軽に使えるため今後に期待したいです。
しかしながら、日本語でやろうとするとダメなケースも散見し、日本語処理を自前で実装すれば解決できるかどうか、今後調査を続けます。
なお、今回の内容は追加修正してどこかでまとめられる予定です。
明日の更新はgg_hatanoさんが担当してくださいます。