Help us understand the problem. What is going on with this article?

Pythonのrequests_futureの実行方法による速度比較

More than 1 year has passed since last update.

Pythonのrequests_futuresの実行方法によって速度が変わる。
requests_futureを使うときは、sessionを一度にまとめて作って、後から評価するようにしないと並列で動かない。多分

テストコード

from concurrent.futures import ThreadPoolExecutor
from requests_futures.sessions import FuturesSession
import requests
import time

session = FuturesSession(executor=ThreadPoolExecutor(max_workers=30))

num_links = 100
links = [f"http://example.com/{_}" for _ in range(1,num_links)]

fs = []
print(f"requesting to {num_links} links")
t1 = time.time()
for l in links:
    f = session.get(l)
    fs.append(f)
t2 = time.time()
print(f"push to pool w/ requests_future: {t2-t1} sec")

for f in fs:
    f.result().status_code == 200
t3 =time.time()
print(f"eval in pool w/ requests_future: {t3-t2} sec")

for l in links:
    f = session.get(l)
    f.result().status_code == 200
t4 = time.time()
print(f"eval everytime w/ requests_future: {t4-t3} sec")

for l in links:
    r = requests.get(l)
    r.status_code == 200
t5 = time.time()
print(f"eval everytime w/ requests: {t5-t5} sec")

実行結果

requesting to 100 links
push to pool w/ requests_future: 0.08985710144042969 sec
eval in pool w/ requests_future: 0.9865047931671143 sec
eval everytime w/ requests_future: 24.10741138458252 sec
eval everytime w/ requests: 44.42911386489868 sec
abeja
「ディープラーニング」を活用し、多様な業界、シーンにおけるビジネスの効率化・自動化を促進するベンチャー企業です。
https://abejainc.com
Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away