LoginSignup
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

【ひとりカレンダー】ClojureAdvent Calendar 2024

Day 25
@maaaashi

Clojure: DuctでRSSリーダーを作る - 完成

Last updated at Posted at 2024-12-24

advent_calendar_2024.png

Advent Calendar 2024 Day 25

昨日は親子関係のデータをDBに入れられるようになりました。

今日はQiitaのRSS受信ができるようにしようと思います

ライブラリのインポート

RSSを受信し、xmlを扱うために幾つかのライブラリを入れていきます

(ns clojure-rss-reader.handler.feeds
  (:require [ataraxy.response :as response]
            [integrant.core :as ig]
            [...]
            [clj-http.client :as http] ; 追加
            [clojure.zip :as zip] ; 追加
            [clojure.data.xml :as xml] ; 追加
            [clojure.data.zip.xml :as zx] ; 追加))
ライブラリ 用途
clj-http.client HTTPリクエストを送信し、レスポンスを受け取る
clojure.zip データ構造（特に木構造）を操作する
clojure.data.xml XMLデータの生成、解析、操作する
clojure.data.zip.xml clojure.zipと組み合わせて、XMLデータを効率的に操作する

ではこれらのライブラリを使ってRSSリーダーを完成させていきます

RSS Readerの実装

(ns clojure-rss-reader.handler.feeds
  (:require [clj-http.client :as http]
            [clojure.zip :as zip]
            [clojure.data.xml :as xml]
            [clojure.data.zip.xml :as zx]))

(defn fetch-rss [url]
  (:body (http/get url {:as :text})))

(defn zipper [rss-content]
  (-> rss-content
      (xml/parse-str)
      (zip/xml-zip)))

(defn extract-articles [entry]
  (let [title (zx/xml1-> entry :title zx/text)
        url (zx/xml1-> entry :url zx/text)
        content (zx/xml1-> entry :content zx/text)
        author (zx/xml1-> entry :author zx/text)
        published (zx/xml1-> entry :published zx/text)]
    {:title title
     :url url
     :content content
     :author author
     :published published}))

(defn extract-feed [zipper]
  (let [title (zx/xml1-> zipper :title zx/text)
        description (zx/xml1-> zipper :description zx/text)
        entries (zx/xml-> zipper :entry)
        articles (map extract-articles entries)]
    {:title title :description description :articles articles}))

こんな感じで実装しました

少しだけ解説をします

fetch-rss

(defn fetch-rss [url]
  (:body (http/get url {:as :text})))

これはclj-http.clientを使って、与えられたurlから、レスポンステキストを取得する関数です

zipper

(defn zipper [rss-content]
  (-> rss-content
      (xml/parse-str)
      (zip/xml-zip)))

これはxmlのテキストを受け取り、
clojure.data.xmlのparse-strを使ってClojureのデータ構造に変換し、
さらにclojure.zipのxml-zipを使ってXMLのツリー構造に変換する関数です

extract-articles

(defn extract-articles [entry]
  (let [title (zx/xml1-> entry :title zx/text)
        url (zx/xml1-> entry :url zx/text)
        content (zx/xml1-> entry :content zx/text)
        author (zx/xml1-> entry :author zx/text)
        published (zx/xml1-> entry :published zx/text)]
    {:title title
     :url url
     :content content
     :author author
     :published published}))

これは xmlデータから、記事の情報である、
title, url, content, author, publishedを抽出する関数です
抽出にはclojure.data.zip.xmlのxml1->を使っています

extract-feed

(defn extract-feed [zipper]
  (let [title (zx/xml1-> zipper :title zx/text)
        description (zx/xml1-> zipper :description zx/text)
        entries (zx/xml-> zipper :entry)
        articles (map extract-articles entries)]
    {:title title :description description :articles articles}))

これは、xmlデータからデータを抽出する関数です
title, descriptionに関してはclojure.data.zip.xmlのxml1->を使い1件抽出し、
entryに関してはclojure.data.zip.xmlのxml->を使ってN件取得しています

N件取得できたentryをmapで先ほどのextract-articlesの関数にそれぞれ渡してarticlesを作成しています

これらと昨日までのDBアクセスを組み合わせて、RSS Readerを完成させてみます

RSS Reader完成

handler/feeds.clj
(ns clojure-rss-reader.handler.feeds
  (:require [ataraxy.response :as response]
            [integrant.core :as ig]
            [next.jdbc :as jdbc]
            [next.jdbc.result-set :as rs]
            [honey.sql :as sql]
            [clj-http.client :as http]
            [clojure.zip :as zip]
            [clojure.data.xml :as xml]
            [clojure.data.zip.xml :as zx]))

(def select-feeds (sql/format {:select [:*]
                               :from [:feed]}))

(defn select-articles [{:keys [id]}]
  (sql/format {:select [:*]
               :from [:article]
               :where [:= :feed_id id]}))

(defn get-feeds [db]
  (let [feeds (jdbc/execute! db select-feeds {:builder-fn rs/as-unqualified-lower-maps})
        feeds-articles (map #(->> (jdbc/execute! db (select-articles %) {:builder-fn rs/as-unqualified-lower-maps})
                                  (assoc % :articles)) feeds)]
    {:feeds feeds-articles}))

(defmethod ig/init-key ::get [_ {{:keys [spec]} :db}]
  (fn [_]
    [::response/ok (get-feeds spec)]))


(defn fetch-rss [url]
  (:body (http/get url {:as :text})))

(defn zipper [rss-content]
  (-> rss-content
      (xml/parse-str)
      (zip/xml-zip)))

(defn extract-articles [entry]
  (let [title (zx/xml1-> entry :title zx/text)
        url (zx/xml1-> entry :url zx/text)
        content (zx/xml1-> entry :content zx/text)
        author (zx/xml1-> entry :author zx/text)
        publicshed (zx/xml1-> entry :published zx/text)]
    {:title title
     :url url
     :content content
     :author author
     :published publicshed}))

(defn extract-feed [zipper]
  (let [title (zx/xml1-> zipper :title zx/text)
        description (zx/xml1-> zipper :description zx/text)
        entries (zx/xml-> zipper :entry)
        articles (map extract-articles entries)]
    {:title title :description description :articles articles}))

(defn insert-article [{:keys [feed_id title url content author published]}]
  (sql/format {:insert-into
               :article
               :columns [:feed_id :title :url :content :author :published_at]
               :values [{:feed_id feed_id :title title :url url :content content :author author :published_at published}]}))

(defn register-articles [db {:keys [feed_id articles]}]
  (doall (map #(jdbc/execute! db (insert-article (assoc % :feed_id feed_id))) articles)))

(defn insert-feed [{:keys [url title description]}]
  (sql/format {:insert-into
               :feed
               :columns [:url :title :description]
               :values [{:url url :title title :description description}]
               :returning [:id]}))

(defn register-feed [db feed-data]
  (let [feed-ids (jdbc/execute! db (insert-feed feed-data) {:builder-fn rs/as-unqualified-lower-maps})
        id (-> feed-ids first :id)]
    (register-articles db (assoc feed-data :feed_id id))))

(defn post-feeds [db {{:keys [url]} :body-params}]
  (let [c (fetch-rss url)
        z (zipper c)
        f (extract-feed z)
        data (assoc f :url url)]
    (register-feed db data))
  [::response/ok {:message "OK"}])

(defmethod ig/init-key ::post [_ {{:keys [spec]} :db}]
  (partial post-feeds spec))

POST: /v1/feedsに受信したいRSSのURLを送り、
GET: /v1/feedsで情報を取得できます

実際にリクエストを送ってみます

$ curl -X POST http://localhost:3000/v1/feeds -H "Content-Type: application/json" -d '{ "url": "https://qiita.com/tags/clojure/feed" }'
{"message":"OK"}

$ curl http://localhost:3000/v1/feeds       
{"feeds":[{"id":1,"url":"https://qiita.com/tags/clojure/feed","title":"Clojureタグが付けられた新着記事 - Qiita","description":"QiitaでClojureタグが付けられた新着記事","articles":[{"id":1,"feed_id":1,"title":"Clojure: DuctでRSSリーダーを作る - HoneySQLその２","url":"https://qiita.com/maaaashi/items/07d5ba7148a5cbe19637","content":" Advent Calendar 2024 Day 24 昨日はHoneySQLを導入し、SQLクエリをClojureのデータ構造（マップやベクター）として構築できるようになりました 今日は親子関係…","author":"maaaashi","published_at":"2024-12-24T07:03:58+09:00"},{"id":2,"feed_id":1,"title":"Clojure: DuctでRSSリーダーを作る - HoneySQL","url":"https://qiita.com/maaaashi/items/5d9f5d8ced8c5f0ecad5","content":" Advent Calendar 2024 Day 23 昨日は環境変数を扱いました。 今日はよりDBからのデータ取得周りを改善してみます HoneySQLというSQLクエリをClojureのデータ…","author":"maaaashi","published_at":"2024-12-23T07:04:04+09:00"},{"id":3,"feed_id":1,"title":"Clojure: DuctでRSSリーダーを作る - 環境変数","url":"https://qiita.com/maaaashi/items/525d676aba72872aac19","content":" Advent Calendar 2024 Day 22 昨日は以下のようにDB接続を設定しました。 :duct.database.sql/hikaricp {:adapter \"postgresq…","author":"maaaashi","published_at":"2024-12-22T07:03:53+09:00"},{"id":4,"feed_id":1,"title":"Clojure: DuctでRSSリーダーを作る - DB接続(HikariCP, Jdbc)","url":"https://qiita.com/maaaashi/items/32fee6a9b924d31e4c9a","content":" Advent Calendar 2024 Day 21 昨日はDuctでルーティングを実装しました 今日はDB接続したいと思います。 今回の記事ではpostgresqlに接続しますが、 DBの立ち…","author":"maaaashi","published_at":"2024-12-21T23:45:41+09:00"}]}]}

いい感じにclojureのtagがついてqiitaの記事が保存・取得できました

これで簡単にですがRSS Readerの完成です

1
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?