8
7

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

日向坂のブログ画像のスクレイピング

Last updated at Posted at 2019-12-24

はじめに

コードに興味がなく、画像が欲しい人はこちら

日向坂のブログのスクレイピングです
今回は画像のダウンロードのみにフォーカスしています

開発環境

Python:3.7
beautifulsoup4:4.8.1

##コード
改善点、ツッコミ等ありましたらコメント or Twitter(@Azumi_cpa)にお願いします

from bs4 import BeautifulSoup
import requests
import time


def get_picture_url(url):
    pic_urls = []
    page_number = 0

    # 記事が表示されなくなるまで各ページの画像のurlを取得
    while True:
        print(url + "&page=" + str(page_number))
        response = requests.get(url + "&page=" + str(page_number))
        soup = BeautifulSoup(response.text, 'lxml')
        a = soup.find_all('div', class_='p-blog-article')

        # ここもっときれいに書けそう
        if a != []:
            for b in a:
                for c in b.find_all('img'):
                    pic_urls.append(c["src"])
            print(page_number)
            page_number += 1
            time.sleep(3)
        else:
            break

    return pic_urls


def save_pictures(name, url):
    # 画像のurl取得
    pic_urls = get_picture_url(url)

    # 保存開始
    for i, url in enumerate(pic_urls):
        try:
            response = requests.get(url)
            image = response.content

            # メンバー名/番号.jpg
            file_name = name + "/" + str(i) + ".jpg"
            with open(file_name, "wb") as aaa:
                aaa.write(image)

            time.sleep(3)
        except:
            print("エラー")


def get_members():
    member_list = {}
    response = requests.get('https://www.hinatazaka46.com/s/official/diary/member?ima=0000')
    soup = BeautifulSoup(response.text, 'lxml')

    members = soup.find_all("a", class_="p-blog-face__list")

    for member in members:
        # スペースと改行を削除
        member_name = member.text.replace(" ", "")
        member_name = member_name.replace("\n", "")

        # ブログの0ページ目のurlも取得(1ページ以降はこれに"&page=n"を追加
        member_list[member_name] = "https://www.hinatazaka46.com" + member.attrs["href"]

    return member_list


def main():
    # メンバーリスト作成
    members_list = get_members()
    
    for name, url in members_list.items():
        print(name + "開始")
        save_pictures(name, url)


main()

今後やること

  • 保存するとき、メンバー名/ブログ日付_n.jpgにする
  • 前回保存分~実行時までの差分のみを保存するようにする
8
7
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
8
7

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?