LoginSignup
2
1
AUTOSAR CountdownAdvent Calendar 2022Day 8
@kaizen_nagoya(Dr. Kiyoshi Ogawa)

Qiita(11)「Qiitaいいな〜自分のいいね分析」自分版

Last updated at Posted at 2019-02-25

Qiitaいいな〜自分のいいね分析
https://qiita.com/snona/items/35e7fde3596d0e9a6bd4

idを自分にして動かそうとしてみた。

$ ruby qiita.rb
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- selenium-webdriver (LoadError)
	from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from qiita.rb:1:in `<main>'

Selenium WebDriver入門
https://qiita.com/yulily@github/items/c3edbe25e84280c17776

$ brew install chromedriver
Error: No available formula with the name "chromedriver" 
It was migrated from homebrew/core to homebrew/cask.
You can access it again by running:
  brew tap homebrew/cask
And then you can install it by running:
  brew cask install chromedriver

$ brew install selenium
Updating Homebrew...
Error: No available formula with the name "selenium" 
==> Searching for a previously deleted formula (in the last month)...
Error: No previously deleted formula found.
==> Searching for similarly named formulae...
This similarly named formula was found:
selenium-server-standalone
To install it, run:
  brew install selenium-server-standalone
==> Searching taps...
==> Searching taps on GitHub...
Error: No formulae found in taps.

$ brew install selenium-server-standalone
Updating Homebrew...

==> Downloading https://selenium-release.storage.googleapis.com/3.141/selenium-server-standalone-3.141.59.jar
######################################################################## 100.0%
==> Caveats
To have launchd start selenium-server-standalone now and restart at login:
  brew services start selenium-server-standalone
Or, if you don't want/need a background service you can just run:
  selenium-server -port 4444
==> Summary
🍺  /usr/local/Cellar/selenium-server-standalone/3.141.59: 5 files, 10.2MB, built in 12 seconds

$ ruby qiita.rb
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- selenium-webdriver (LoadError)
	from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from qiita.rb:1:in `<main>'



$  brew cask install chromedriver
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
get_iplayer                                                                     ntopng

==> Satisfying dependencies
==> Downloading https://chromedriver.storage.googleapis.com/2.46/chromedriver_mac64.zip
######################################################################## 100.0%
==> Verifying SHA-256 checksum for Cask 'chromedriver'.
==> Installing Cask chromedriver
==> Linking Binary 'chromedriver' to '/usr/local/bin/chromedriver'.
🍺  chromedriver was successfully installed!

$ ruby qiita.rb
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- selenium-webdriver (LoadError)
	from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from qiita.rb:1:in `<main>'

「cannot load such file -- selenium-webdriver (LoadError)」検索

gemファイル　cannot load such file
https://teratail.com/questions/118997

$ gem install selenium-webdriver
Fetching: rubyzip-1.2.2.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /Library/Ruby/Gems/2.3.0 directory.

$ sudo gem install selenium-webdriver
Password:
Fetching: rubyzip-1.2.2.gem (100%)
Successfully installed rubyzip-1.2.2
Fetching: ffi-1.10.0.gem (100%)
Building native extensions.  This could take a while...
Successfully installed ffi-1.10.0
Fetching: childprocess-0.9.0.gem (100%)
Successfully installed childprocess-0.9.0
Fetching: selenium-webdriver-3.141.0.gem (100%)
Successfully installed selenium-webdriver-3.141.0
Parsing documentation for rubyzip-1.2.2
Installing ri documentation for rubyzip-1.2.2
Parsing documentation for ffi-1.10.0
Installing ri documentation for ffi-1.10.0
Parsing documentation for childprocess-0.9.0
Installing ri documentation for childprocess-0.9.0
Parsing documentation for selenium-webdriver-3.141.0
Installing ri documentation for selenium-webdriver-3.141.0
Done installing documentation for rubyzip, ffi, childprocess, selenium-webdriver after 49 seconds
4 gems installed

$ ruby qiita.rb

ようやく動作し始めました。
100ページで5分かかっています。

処理が終わり、上位１００件UP

tag 件数
Python 359
マルチコア 149
JSX 147
tweepy 143
TwitterAPI 143
古文 142
docker 133
機械学習 123
言語処理100本ノック 120
Linux 119
C++ 109
JavaScript 97
C 84
TOPPERS 78
RaspberryPi 69
Python3 66
自然言語処理 65
Coq 65
DeepLearning 64
量子コンピュータ 61
Mac 56
Java 54
MIDI 54
Arduino 52
FPGA 50
アセンブラ 50
Git 49
ETロボコン 49
Swift 48
R 46
LaTeX 45
Qiita 44
OCaml 41
Ruby 40
Verilog 38
iOS 38
EV3RT 37
Windows 37
初心者 36
cobol 36
GitHub 35
assembly 33
Ubuntu 33
x86 33
IoT 33
新人プログラマ応援 32
ARM 31
Xcode 29
EV3 28
bioinformatics 28
TensorFlow 28
gdb 28
SystemVerilog 28
PHP 28
CPU 27
ECHONET 27
LLVM 27
Haskell 26
ポエム 26
MacOSX 26
mecab 26
mruby 25
Bash 24
curl 24
深層学習 24
GPU 24
C# 23
音楽 23
Z80 23
GCC 22
Android 22
SoftEther 22
C言語 22
Raspberrypi3 22
HDL 21
RTOS 21
色 21
VPN 21
athrill 21
word2vec 21
プログラミング 20
ssreflect 20
WebAssembly 19
Wireshark 19
AlloyAnalyzer 18
量子ゲート 18
Alloy 18
セキュリティ 18
Bioconductor 18
CentOS 18
CMake 17
LilyPond 17
GitLab 17
AWS 17
おうちハック 17
Eclipse 16
rfc 16
R2CA 16
kernel 16
Scala 16

JSX , tweepy, TwitterAPI, 古文の３位から６位までの件数がおかしい。ひょっとしたらマルチコアも。
原因・理由は不明。それ以外は、やや少ない気がするものはある程度。
はっきりおかしいものは上記の４つか５つ。

どうやって調べるといいかを調べ中。

いいね数 件数
1 324
2 311
3 280
4 211
5 171
161 144
280 143
461 142
272 142
1237 142
6 137
7 105
8 88
9 81
10 63
11 59
12 54
14 52
13 42
15 35

おお、こちらは、161, 280, 461, 272 1237の５つがおかしい。
Rubyのデバッグ方法を調査。

調査

ソースの表示

Qiitaいいな〜自分のいいね分析
https://qiita.com/snona/items/35e7fde3596d0e9a6bd4
のソースコードのIDを変更しただけのもの。

qiita.rb
require 'selenium-webdriver'

# 対象ユーザのQiitaID
id = 'kaizen_japan'

driver = Selenium::WebDriver.for :chrome

# いいねの記事一覧ページに移動
driver.navigate.to "https://qiita.com/#{id}/like"

# いいね, タグの集計
likes = Hash.new(0)
tags = Hash.new(0)

loop do
  # いいね数を取得
  driver.find_elements(:class, 'fa-like').each{|like| likes[like.find_element(:xpath, '..').text] += 1}

  # タグを取得
  driver.find_elements(:class, 'TagList__label').each{|tag| tags[tag.text] += 1}

  # 次ページへ遷移
  begin
    driver.find_element(:class, 'js-next-page-link').click
  rescue
    # 見つからない場合は終了
    break
  end
end

driver.quit

# いいねの結果を出力
puts 'likes'
likes.sort{|(k1, v1), (k2, v2)| v2 <=> v1}.each{|key, value| puts "#{key},#{value}"}

# タグの集計を出力
puts 'tags'
tags.sort{|(k1, v1), (k2, v2)| v2 <=> v1}.each{|key, value| puts "#{key},#{value}"}

##わかった行

いいねの記事一覧ページに移動
?driver.navigate.to "https://qiita.com/#{id}/like"

１ページごとにブラウザに表示している。できれば、画面の表示は省略したいが、方法をまだ知らない。

次ページへ遷移
begin
driver.find_element(:class, 'js-next-page-link').click
rescue

次のページに遷移する。今、どのページを処理しているかは分かる。

#結果

@tag1216 さんからいただいたコメントで、それぞれ左側の値も加算していることが原因であることが判明。

ひとまず、71ページ分の値を減算した結果

tag 件数
Python 359
docker 133
機械学習 123
言語処理100本ノック 120
Linux 119
C++ 109
JavaScript 97
C 84
TOPPERS 78
RaspberryPi 69
Python3 66
Coq 65
自然言語処理 65
DeepLearning 64
量子コンピュータ 61
Mac 56
Java 54
MIDI 54
Arduino 52
FPGA 50
アセンブラ 50
ETロボコン 49
Git 49
Swift 48
R 46
LaTeX 45
Qiita 44
OCaml 41
Ruby 40
iOS 38
Verilog 38
EV3RT 37
Windows 37
cobol 36
初心者 36
GitHub 35
assembly 33
IoT 33
Ubuntu 33
x86 33
新人プログラマ応援 32
ARM 31
Xcode 29
bioinformatics 28
EV3 28
gdb 28
PHP 28
SystemVerilog 28
TensorFlow 28
CPU 27
ECHONET 27
LLVM 27
Haskell 26
MacOSX 26
mecab 26
ポエム 26
mruby 25
Bash 24
curl 24
GPU 24
深層学習 24
C# 23
Z80 23
音楽 23
Android 22
C言語 22
GCC 22
Raspberrypi3 22
SoftEther 22
athrill 21
HDL 21
RTOS 21
VPN 21
word2vec 21
色 21
ssreflect 20
プログラミング 20
WebAssembly 19
Wireshark 19
Alloy 18
AlloyAnalyzer 18
Bioconductor 18
CentOS 18
セキュリティ 18
量子ゲート 18
AWS 17
CMake 17
GitLab 17
LilyPond 17
おうちハック 17
Eclipse 16
kernel 16
R2CA 16
rfc 16
Scala 16
電子工作 16
Node.js 16
MSX 16
UNIX 16
Home@KAIT 16
いいね 件数
1 324
2 311
3 280
4 211
5 171
6 137
7 105
8 88
9 81
10 63
11 59
12 54
14 52
13 42
15 35
16 34
18 23
22 23
23 23
20 22
19 19
17 17
25 16
33 16
21 14
29 13
30 13
24 12
27 12
28 12
39 12
26 11
38 11
36 10
41 10
32 9
35 9
46 9
56 9
37 8
48 8
31 7
40 7
42 7
45 7
63 7
83 7
34 6
43 6
44 6
53 6
55 6
96 6
50 5
60 5
61 5
112 5
52 4
54 4
71 4
73 4
87 4
94 4
95 4
109 4
189 4
15項目 3
37項目 2
198項目 1

2020時点の不具合

macOS
$ ruby qiita.rb
/Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/response.rb:69:in `assert_ok': session not created: Chrome version must be between 71 and 75 (Selenium::WebDriver::Error::SessionNotCreatedError)
  (Driver info: chromedriver=2.46.628411 (3324f4c8be9ff2f70a05a30ebc72ffb013e1a71e),platform=Mac OS X 10.13.6 x86_64)
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/response.rb:32:in `initialize'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/http/common.rb:84:in `new'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/http/common.rb:84:in `create_response'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/http/default.rb:104:in `request'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/http/common.rb:62:in `call'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/bridge.rb:166:in `execute'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/bridge.rb:99:in `create_session'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/bridge.rb:53:in `handshake'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/chrome/driver.rb:49:in `initialize'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/common/driver.rb:44:in `new'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/common/driver.rb:44:in `for'
	from /Library/Ruby/Gems/2.3.0/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver.rb:86:in `for'
	from qiita.rb:6:in `<main>'

文書履歴(document history)

ver. 0.01 初稿 20190225 昼
ver. 0.02 ソースの分かる行を確認 20190225 午後４時
ver. 0.03 ひとまず手で引き算 20190225 夕
ver. 0.04 2020年時点でのエラー追記 20200517

最後までおよみいただきありがとうございました。

いいね　💚、フォローをお願いします。

Thank you very much for reading to the last sentence.

Please press the like icon 💚　and follow me for your happy life.

2
1
2

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
2
1