More than 5 years have passed since last update.

Rubyクローラーでユーザーエージェントを指定する

Last updated at 2015-01-27Posted at 2015-01-27

参考文献
https://github.com/ZIGExN/ruby-trailning/wiki/Web%E3%82%B9%E3%82%AF%E3%83%AC%E3%82%A4%E3%83%94%E3%83%B3%E3%82%B0%E3%82%92%E7%94%A8%E3%81%84%E3%81%9F%E3%83%87%E3%83%BC%E3%82%BF%E6%A4%9C%E7%B4%A2%E3%82%AF%E3%83%AD%E3%83%BC%E3%83%A9%E6%A7%8B%E7%AF%89%E5%85%A5%E9%96%80

Google検索の仕組み
https://support.google.com/webmasters/answer/70897?hl=ja

Google

Googleクローラー

# (例)
html = open(url, 'User-Agent' => 'Googlebot/2.1')
doc = Nokogiri::HTML.parse(html, nil, "UTF-8")

Yahooクローラー

user_agent = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.63 Safari/537.36'
html = open(url, "User-Agent" => user_agent

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up