8

More than 5 years have passed since last update.

@Kenya(ブライアン)

Rubyクローラーでユーザーエージェントを指定する

Last updated at 2015-01-27Posted at 2015-01-27

参考文献
https://github.com/ZIGExN/ruby-trailning/wiki/Web%E3%82%B9%E3%82%AF%E3%83%AC%E3%82%A4%E3%83%94%E3%83%B3%E3%82%B0%E3%82%92%E7%94%A8%E3%81%84%E3%81%9F%E3%83%87%E3%83%BC%E3%82%BF%E6%A4%9C%E7%B4%A2%E3%82%AF%E3%83%AD%E3%83%BC%E3%83%A9%E6%A7%8B%E7%AF%89%E5%85%A5%E9%96%80

Google検索の仕組み
https://support.google.com/webmasters/answer/70897?hl=ja

Google

Googleクローラー

#(例)
html = open(url, 'User-Agent' => 'Googlebot/2.1')
doc = Nokogiri::HTML.parse(html, nil, "UTF-8")

Yahooクローラー

user_agent = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.63 Safari/537.36'
html = open(url, "User-Agent" => user_agent

8

Register as a new user and use Qiita more conveniently

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up

8