Go to Qiita Advent Calendar 2024 Top
LoginSignup
6
4

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 1 year has passed since last update.

@yu-chan_101

【Python3】Selenium3⇒4にしたらProxy環境下で動作しなくなった話(解決)

Last updated at Posted at 2022-10-18

概要

社内Proxy環境下、selenium3で作成し実行していたスクリプトを、【Python】Selenium 4の変更点とWarning(zke様)を参考にselenium4の記述で書き直したところ、driver起動の部分でエラー発生
調べても出てこず困っていましたが、解決したので方法を残しておく

環境

python 3.9.13
selenium 4.5.0

要約

・環境変数no_proxyが設定されていないとlocalhostにプロキシ通そうとするコードに書き換わっていた
・環境変数no_proxyに、localhost,127.0.0.1の値を設定したら動くようになった
(実行したcmd、エディタ等の再起動必須)

エラー内容

ソースコード

from selenium import webdriver
from webdriver_manager.microsoft import IEDriverManager
from selenium.webdriver.ie.service import Service

options = webdriver.IeOptions()
options.add_additional_option("ie.edgechromium", True)
options.add_additional_option("detach", True)
options.add_additional_option(
    "ie.edgepath", r'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe')


driver = webdriver.Ie(
    service=Service(executable_path=IEDriverManager().install()), options=options)

エラー

Traceback (most recent call last):
  File "***.py", line 9, in __init__
    driver = webdriver.Ie(
  File "c:\Users\仮想環境のパス\lib\site-packages\selenium\webdriver\ie\webdriver.py", line 110, in __init__
    super().__init__(
  File "c:\Users\仮想環境のパス\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 272, in __init__
    self.start_session(capabilities, browser_profile)
  File "c:\Users\仮想環境のパス\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 364, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "c:\Users\仮想環境のパス\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 429, in execute
    self.error_handler.check_response(response)
  File "c:\Users\仮想環境のパス\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 207, in check_response
    raise exception_class(value)
selenium.common.exceptions.WebDriverException: Message: <html><head>
<meta type="copyright" content="Copyright (C) 1996-2018 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>エラー: 要求されたURLからデータを取り出せませんでした。</title>
<style type="text/css"><!--
 /*
 * Copyright (C) 1996-2018 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.
 */

/*
 Stylesheet for Squid Error pages
 Adapted from design by Free CSS Templates
 http://www.freecsstemplates.org
 Released for free under a Creative Commons Attribution 2.5 License
*/

/* Page basics */
* {
        font-family: verdana, sans-serif;
}

html body {
        margin: 0;
        padding: 0;
        background: #efefef;
        font-size: 12px;
        color: #1e1e1e;
}

/* Page displayed title area */
#titles {
        margin-left: 15px;
        padding: 10px;
        padding-left: 100px;
        background: url('/squid-internal-static/icons/SN.png') no-repeat left;
}

/* initial title */
#titles h1 {
        color: #000000;
}
#titles h2 {
        color: #000000;
}

/* special event: FTP success page titles */
#titles ftpsuccess {
        background-color:#00ff00;
        width:100%;
}

/* Page displayed body content area */
#content {
        padding: 10px;
        background: #ffffff;
}

/* General text */
p {
}

/* error brief description */
#error p {
}

/* some data which may have caused the problem */
#data {
}

/* the error message received from the system or other software */
#sysmsg {
}

pre {
    font-family:sans-serif;
}

/* special event: FTP / Gopher directory listing */
#dirmsg {
    font-family: courier;
    color: black;
    font-size: 10pt;
}
#dirlisting {
    margin-left: 2%;
    margin-right: 2%;
}
#dirlisting tr.entry td.icon,td.filename,td.size,td.date {
    border-bottom: groove;
}
#dirlisting td.size {
    width: 50px;
    text-align: right;
    padding-right: 5px;
}

/* horizontal lines */
hr {
        margin: 0;
}

/* page displayed footer area */
#footer {
        font-size: 9px;
        padding-left: 10px;
}


body
:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }
:lang(he) { direction: rtl; }
 --></style>
</head><body id="ERR_CONNECT_FAIL">
<div id="titles">
<h1>エラー</h1>
<h2>指定された URL を取得できませんでした</h2>
</div>
<hr>

<div id="content">
<p>URL: <a href="http://localhost:53934/session">http://localhost:53934/session</a> の取得中に以下のエラーが発生しました。</p>

<blockquote id="error">
<p><b>127.0.0.1 に接続できませんでした。</b></p>
</blockquote>

<p id="sysmsg">システムが以下のエラーを返しました: <i>(111) Connection refused</i></p>

<p>The remote host or network may be down. Please try the request again.</p>

<p>Your cache administrator is <a href="mailto:webmaster?subject=CacheErrorInfo%20-%20ERR_CONNECT_FAIL&amp;body=CacheHost%3A%20adnpxin10h%0D%0AErrPage%3A%20ERR_CONNECT_FAIL%0D%0AErr%3A%20(111)%20Connection%20refused%0D%0ATimeStamp%3A%20Tue,%2018%20Oct%202022%2001%3A31%3A21%20GMT%0D%0A%0D%0AClientIP%3A%2010.13.164.32%0D%0AServerIP%3A%20localhost%0D%0A%0D%0AHTTP%20Request%3A%0D%0APOST%20%2Fsession%20HTTP%2F1.1%0AAccept-Encoding%3A%20identity%0D%0AContent-Length%3A%20291%0D%0AAccept%3A%20application%2Fjson%0D%0AContent-Type%3A%20application%2Fjson%3Bcharset%3DUTF-8%0D%0AUser-Agent%3A%20selenium%2F4.5.0%20(python%20windows)%0D%0AConnection%3A%20keep-alive%0D%0AHost%3A%20localhost%3A53934%0D%0A%0D%0A%0D%0A">webmaster</a>.</p>

<br>
</div>

<hr>
<div id="footer">
<p>Tue, 18 Oct 2022 01:31:21 GMTにadnpxin10h (squid/3.5.28)が生成しました。</p>
<!-- ERR_CONNECT_FAIL -->
</div>
</body></html>

・HTML部分を保存して開いた結果

image.png

原因と解決策

原因

selenium/webdriver/remote/remote_connection.py
    def __init__(self, remote_server_addr, keep_alive=False, ignore_proxy: typing.Optional[bool] = False):
        self.keep_alive = keep_alive
        self._url = remote_server_addr

        # Env var NO_PROXY will override this part of the code
        _no_proxy = os.environ.get('no_proxy', os.environ.get('NO_PROXY'))
        if _no_proxy:
            for npu in _no_proxy.split(','):
                npu = npu.strip()
                if npu == "*":
                    ignore_proxy = True
                    break
                n_url = parse.urlparse(npu)
                remote_add = parse.urlparse(self._url)
                if n_url.netloc:
                    if remote_add.netloc == n_url.netloc:
                        ignore_proxy = True
                        break
                else:
                    if n_url.path in remote_add.netloc:
                        ignore_proxy = True
                        break

        self._proxy_url = self._get_proxy_url() if not ignore_proxy else None
        if keep_alive:
            self._conn = self._get_connection_manager()

環境変数no_proxyを参照して、ドライバ起動時のlocalhost接続時にProxyを通すか決定するように仕様が変わっていた

解決策

環境変数no_proxyに、localhost,127.0.0.1の値を設定
新しいcmdを起動して実行すると動作するように

環境変数に登録すると困る場合はpythonスクリプト上で環境変数の追加⇒実行後削除をするとよいかも？

6
4
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
6
4

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?