More than 5 years have passed since last update.

railsでcrawler、botのアクセスを制御する

Ruby

Last updated at 2013-07-06Posted at 2013-07-06

is_crawler.rb

# -*- coding: utf-8 -*-
def crawler? ua
  crawler_ua = [
    "crawler",
    "bot",
    "spider",
    "YahooFeedSeeker",
    "Y!J-(?:BSC|BRO|SRD)/",
    "internal +dummy +connection",
    "libwww",
    "HyperEstraier",
    "curl",
    "wget",
    "facebookexternalhit",
    "ia_archiver",
    "Tumblr"
  ]

  crawler_ua.each{ |reg|
    return true if (/#{reg}/i =~ ua) != nil
  }
  return false
end

p crawler?(request.user_agent)
p crawler?("Y!J-BSC/YATS crawler")
p crawler?('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5')

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up