2
4

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

robots.txtを解析するパッケージを作った(python)

Last updated at Posted at 2021-09-05

robots.txtって何

webサイト側で「ここにはアクセスしちゃ駄目だよ〜」等の指示をクローラーに向けて書いたものです
なので、サイトをクローリングするときは最小限守ったほうがいいと思います

どうやって解析するの

まずはurllibを使う方法です(今回はgoogleを例とします)
urllib.robotparser.RobotFileParserクラスを使います

from urllib.robotparser import RobotFileParser

url = "https://www.google.com/robots.txt"
rp = RobotFileParser() # RobotFileParserインスタンスを生成
rp.set_url(url)
rp.read() # robots.txtの読み取り
rp.can_fetch("アクセスしたいURL") # 指定されたURLがアクセス可能か確認、戻り値はbool

基本的な使い方はこんな感じです
これの面倒くさいところはrp.set_urlにrobots.txtの直リンを与えないといけないところです

robots.txtの直リンはurllib.parse.urljoin(url, "/robots.txt")で得られるのですがいちいちこんなのやるの面倒くさいじゃないですか

更にrobots.txtの指示を各User-agentごとに見たい!なんて思ってる方もいるかもしれません

そんなときにはrobotsparsetoolsをお使いください

robotsparsetools

インストールはpipで出来ます

$ pip install robotsparsetools

使い方は簡単、robotsparsetools.Parseにurlを渡してあげるだけです

from robotsparsetools import Parse

url = "https://www.google.com/" # robots.txtへの直リンを渡してもいい
p = Parse(url)
print(p)

これを実行すると、User-agentをkeyとしてそれに対する各指示をvalueとしたdict(正確には違うけど)が返ってきます

出力
{'*': {'Disallow': ['/search', '/sdch', '/groups', '/index.html?', '/?', '/?hl=*&', '/?hl=*&*&gws_rd=ssl', '/imgres', '/u/', '/preferences', '/setprefs', '/default', '/m?', '/m/', '/wml?', '/wml/?', '/wml/search?', '/xhtml?', '/xhtml/?', '/xhtml/search?', '/xml?', '/imode?', '/imode/?', '/imode/search?', '/jsky?', '/jsky/?', '/jsky/search?', '/pda?', '/pda/?', '/pda/search?', '/sprint_xhtml', '/sprint_wml', '/pqa', '/palm', '/gwt/', '/purchases', '/local?', '/local_url', '/shihui?', '/shihui/', '/products?', '/product_', '/products_', '/products;', '/print', '/books/', '/bkshp?*q=*', '/books?*q=*', '/books?*output=*', '/books?*pg=*', '/books?*jtp=*', '/books?*jscmd=*', '/books?*buy=*', '/books?*zoom=*', '/ebooks/', '/ebooks?*q=*', '/ebooks?*output=*', '/ebooks?*pg=*', '/ebooks?*jscmd=*', '/ebooks?*buy=*', '/ebooks?*zoom=*', '/patents?', '/patents/download/', '/patents/pdf/', '/patents/related/', '/scholar', '/citations?', '/citations?*cstart=', '/s?', '/maps?', '/mapstt?', '/mapslt?', '/maps/stk/', '/maps/br?', '/mapabcpoi?', '/maphp?', '/mapprint?', '/maps/api/js/', '/maps/api/place/js/', '/maps/api/staticmap', '/maps/api/streetview', '/maps/_/sw/manifest.json', '/mld?', '/staticmap?', '/maps/preview', '/maps/place', '/maps/timeline/', '/help/maps/streetview/partners/welcome/', '/help/maps/indoormaps/partners/', '/lochp?', '/center', '/ie?', '/blogsearch/', '/blogsearch_feeds', '/advanced_blog_search', '/uds/', '/chart?', '/transit?', '/calendar/', '/cl2/feeds/', '/cl2/ical/', '/coop/directory', '/coop/manage', '/trends?', '/trends/music?', '/trends/hottrends?', '/trends/viz?', '/trends/embed.js?', '/trends/fetchComponent?', '/trends/beta', '/trends/topics', '/musica', '/musicad', '/musicas', '/musicl', '/musics', '/musicsearch', '/musicsp', '/musiclp', '/urchin_test/', '/movies?', '/wapsearch?', '/reviews/search?', '/orkut/albums', '/cbk', '/recharge/dashboard/car', '/recharge/dashboard/static/', '/profiles/me', '/s2/profiles/me', '/s2', '/transconsole/portal/', '/gcc/', '/aclk', '/cse?', '/cse/home', '/cse/panel', '/cse/manage', '/tbproxy/', '/imesync/', '/shenghuo/search?', '/support/forum/search?', '/reviews/polls/', '/hosted/images/', '/ppob/?', '/ppob?', '/accounts/ClientLogin', '/accounts/ClientAuth', '/accounts/o8', '/topicsearch?q=', '/xfx7/', '/squared/api', '/squared/search', '/squared/table', '/qnasearch?', '/app/updates', '/sidewiki/entry/', '/quality_form?', '/labs/popgadget/search', '/buzz/post', '/compressiontest/', '/analytics/feeds/', '/analytics/partners/comments/', '/analytics/portal/', '/analytics/uploads/', '/alerts/', '/ads/search?', '/ads/plan/action_plan?', '/ads/plan/api/', '/ads/hotels/partners', '/phone/compare/?', '/travel/clk', '/hotelfinder/rpc', '/hotels/rpc', '/commercesearch/services/', '/evaluation/', '/chrome/browser/mobile/tour', '/compare/*/apply*', '/forms/perks/', '/shopping/suppliers/search', '/ct/', '/edu/cs4hs/', '/trustedstores/s/', '/trustedstores/tm2', '/trustedstores/verify', '/adwords/proposal', '/shopping?*', '/shopping/product/', '/shopping/seller', '/shopping/ratings/account/metrics', '/shopping/ratings/merchant/immersivedetails', '/shopping/reviewer', '/about/careers/applications/', '/landing/signout.html', '/webmasters/sitemaps/ping?', '/ping?', '/gallery/', '/landing/now/ontap/', '/maps/reserve/api/', '/maps/reserve/search', '/maps/reserve/bookings', '/maps/reserve/settings', '/maps/reserve/manage', '/maps/reserve/payment', '/maps/reserve/receipt', '/maps/reserve/sellersignup', '/maps/reserve/payments', '/maps/reserve/feedback', '/maps/reserve/terms', '/maps/reserve/m/', '/maps/reserve/b/', '/maps/reserve/partner-dashboard', '/about/views/', '/intl/*/about/views/', '/local/cars', '/local/cars/', '/local/dealership/', '/local/dining/', '/local/place/products/', '/local/place/reviews/', '/local/place/rap/', '/local/tab/', '/localservices/*', '/nonprofits/account/'], 'Allow': ['/search/about', '/search/static', '/search/howsearchworks', '/?hl=', '/?hl=*&gws_rd=ssl$', '/?gws_rd=ssl$', '/?pt1=true$', '   /m/finance', '/books?*q=related:*', '/books?*q=editions:*', '/books?*q=subject:*', '/books/about', '/booksrightsholders', '/books?*zoom=1*', '/books?*zoom=5*', '/books/content?*zoom=1*', '/books/content?*zoom=5*', '/ebooks?*q=related:*', '/ebooks?*q=editions:*', '/ebooks?*q=subject:*', '/ebooks?*zoom=1*', '/ebooks?*zoom=5*', '/citations?user=', '/citations?view_op=new_profile', '/citations?view_op=top_venues', '/scholar_share', '/maps?*output=classic*', '/maps?*file=', '/maps/d/', '/maps/api/js', '   /calendar$', '   /calendar/about/', '/safebrowsing/diagnostic', '/safebrowsing/report_badware/', '/safebrowsing/report_error/', '/safebrowsing/report_phish/', '/profiles', '/s2/profiles', '/s2/oz', '/s2/photos', '/s2/search/social', '/s2/static', '/accounts/o8/id', '/alerts/manage', '/alerts/remove', '/alerts/$', '/searchhistory/', '/maps/reserve', '/maps/reserve/partners', '/finance', '/js/']}, 'AdsBot-Google': {'Disallow': ['/maps/api/js/', '/maps/api/place/js/', '/maps/api/staticmap', '/maps/api/streetview'], 'Allow': ['/maps/api/js']}, 'Twitterbot': {'Allow': ['/imgres']}, 'facebookexternalhit': {'Allow': ['/imgres']}, 'Sitemap': ['https://www.google.com/sitemap.xml']}

getメソッドなども使用できます

p.get("*").get("Disallow")
p["*"]["Disallow"]
出力
['/search', '/sdch', '/groups', '/index.html?', '/?', '/?hl=*&', '/?hl=*&*&gws_rd=ssl', '/imgres', '/u/', '/preferences', '/setprefs', '/default', '/m?', '/m/', '/wml?', '/wml/?', '/wml/search?', '/xhtml?', '/xhtml/?', '/xhtml/search?', '/xml?', '/imode?', '/imode/?', '/imode/search?', '/jsky?', '/jsky/?', '/jsky/search?', '/pda?', '/pda/?', '/pda/search?', '/sprint_xhtml', '/sprint_wml', '/pqa', '/palm', '/gwt/', '/purchases', '/local?', '/local_url', '/shihui?', '/shihui/', '/products?', '/product_', '/products_', '/products;', '/print', '/books/', '/bkshp?*q=*', '/books?*q=*', '/books?*output=*', '/books?*pg=*', '/books?*jtp=*', '/books?*jscmd=*', '/books?*buy=*', '/books?*zoom=*', '/ebooks/', '/ebooks?*q=*', '/ebooks?*output=*', '/ebooks?*pg=*', '/ebooks?*jscmd=*', '/ebooks?*buy=*', '/ebooks?*zoom=*', '/patents?', '/patents/download/', '/patents/pdf/', '/patents/related/', '/scholar', '/citations?', '/citations?*cstart=', '/s?', '/maps?', '/mapstt?', '/mapslt?', '/maps/stk/', '/maps/br?', '/mapabcpoi?', '/maphp?', '/mapprint?', '/maps/api/js/', '/maps/api/place/js/', '/maps/api/staticmap', '/maps/api/streetview', '/maps/_/sw/manifest.json', '/mld?', '/staticmap?', '/maps/preview', '/maps/place', '/maps/timeline/', '/help/maps/streetview/partners/welcome/', '/help/maps/indoormaps/partners/', '/lochp?', '/center', '/ie?', '/blogsearch/', '/blogsearch_feeds', '/advanced_blog_search', '/uds/', '/chart?', '/transit?', '/calendar/', '/cl2/feeds/', '/cl2/ical/', '/coop/directory', '/coop/manage', '/trends?', '/trends/music?', '/trends/hottrends?', '/trends/viz?', '/trends/embed.js?', '/trends/fetchComponent?', '/trends/beta', '/trends/topics', '/musica', '/musicad', '/musicas', '/musicl', '/musics', '/musicsearch', '/musicsp', '/musiclp', '/urchin_test/', '/movies?', '/wapsearch?', '/reviews/search?', '/orkut/albums', '/cbk', '/recharge/dashboard/car', '/recharge/dashboard/static/', '/profiles/me', '/s2/profiles/me', '/s2', '/transconsole/portal/', '/gcc/', '/aclk', '/cse?', '/cse/home', '/cse/panel', '/cse/manage', '/tbproxy/', '/imesync/', '/shenghuo/search?', '/support/forum/search?', '/reviews/polls/', '/hosted/images/', '/ppob/?', '/ppob?', '/accounts/ClientLogin', '/accounts/ClientAuth', '/accounts/o8', '/topicsearch?q=', '/xfx7/', '/squared/api', '/squared/search', '/squared/table', '/qnasearch?', '/app/updates', '/sidewiki/entry/', '/quality_form?', '/labs/popgadget/search', '/buzz/post', '/compressiontest/', '/analytics/feeds/', '/analytics/partners/comments/', '/analytics/portal/', '/analytics/uploads/', '/alerts/', '/ads/search?', '/ads/plan/action_plan?', '/ads/plan/api/', '/ads/hotels/partners', '/phone/compare/?', '/travel/clk', '/hotelfinder/rpc', '/hotels/rpc', '/commercesearch/services/', '/evaluation/', '/chrome/browser/mobile/tour', '/compare/*/apply*', '/forms/perks/', '/shopping/suppliers/search', '/ct/', '/edu/cs4hs/', '/trustedstores/s/', '/trustedstores/tm2', '/trustedstores/verify', '/adwords/proposal', '/shopping?*', '/shopping/product/', '/shopping/seller', '/shopping/ratings/account/metrics', '/shopping/ratings/merchant/immersivedetails', '/shopping/reviewer', '/about/careers/applications/', '/landing/signout.html', '/webmasters/sitemaps/ping?', '/ping?', '/gallery/', '/landing/now/ontap/', '/maps/reserve/api/', '/maps/reserve/search', '/maps/reserve/bookings', '/maps/reserve/settings', '/maps/reserve/manage', '/maps/reserve/payment', '/maps/reserve/receipt', '/maps/reserve/sellersignup', '/maps/reserve/payments', '/maps/reserve/feedback', '/maps/reserve/terms', '/maps/reserve/m/', '/maps/reserve/b/', '/maps/reserve/partner-dashboard', '/about/views/', '/intl/*/about/views/', '/local/cars', '/local/cars/', '/local/dealership/', '/local/dining/', '/local/place/products/', '/local/place/reviews/', '/local/place/rap/', '/local/tab/', '/localservices/*', '/nonprofits/account/']

URLへのアクセスが禁止されていないか調べるのは、p. can_crawlで調べることができます(これはrp.can_fetchにあたる関数です)

p.can_crawl("調べたいURL", "User-agent名") # 戻り値はbool

第二引数のUser-agent名を省略した場合は、*(全てのクローラー)が指定されたものとして処理します

Crawl-delay(クロール間隔)はp.delay(User-agent名)で取得できます

p.delay("User-agent")

こちらもUser-agent名を省略した場合は、*が指定されたものとして処理します

Sitemapは、p.get("Sitemap")かp["Sitemap"]で取得できます

p.get("Sitemap")
p["Sitemap"]
# どっちでも取得できる

また、あまり使わないかもしれませんが、robots.txtへのアクセスに、urlopenではなくrequestsを使用することもできます
requests.getに渡すオプションも指定できます

p = Parse("https://www.google.com/", requests=True, timeout=(1.0, 10.0))

まとめ

限定的かもしれませんが、是非robots.txtの解析にrobotsparsetoolsを使ってみてください!

2
4
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
2
4

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?