Qiita Teams that are logged in
You are not logged in to any team

Log in to Qiita Team
Community
OrganizationAdvent CalendarQiitadon (β)
Service
Qiita JobsQiita ZineQiita Blog
9
Help us understand the problem. What is going on with this article?
@kou_pg_0131

【TypeScript】Webスクレイピングのやり方

関連

準備

今回使用するモジュールはこちら

  • request, request-promise-native, @types/request-promise-native
    HTTPリクエスト送信
    (2020/06/22 追記)こちらのライブラリは既に deprecated になっているので、他の代替ライブラリの仕様を検討してください。代替ライブラリの例としては「Alternative libraries to request · Issue #3143 · request/request」をご参照ください。

  • chardet, @types/chardet

    文字コード判定

  • iconv-lite

    文字コード変換

  • jsdom, @types/jsdom

    HTMLパーサ

$ npm i request request-promise-native @types/request-promise-native chardet @types/chardet iconv-lite jsdom @types/jsdom

サンプルコード

import * as request from 'request-promise-native';
import * as chardet from 'chardet';
import * as iconv   from 'iconv-lite';
import { JSDOM }    from 'jsdom';

const main = async () => {
  const url = 'スクレイピングしたいページのURL';

  // HTTPリクエスト送信
  const option = {
    url: url,
    encoding: null
  };
  const response = await request.get(option);

  // 文字コード判別
  const encoding = chardet.detect(response)!.toString();
  if(!encoding) throw new Error();

  // 文字コード変換
  const html: string = iconv.decode(response, encoding);

  // HTMLパース
  const jsdom = new JSDOM(html);
  const document = jsdom.window.document;

  // スクレイピング
  // ここからはquerySelectorとかgetElementByIdとかお馴染みのメソッドで好きなだけスクレイピングしちゃってください
  const elm = document.querySelector('hogehoge');

  // ...
};
main();

参考

9
Help us understand the problem. What is going on with this article?
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
kou_pg_0131
Goが好き

Comments

No comments
Sign up for free and join this conversation.
Sign Up
If you already have a Qiita account Login
9
Help us understand the problem. What is going on with this article?