Go to Qiita Advent Calendar 2024 Top
LoginSignup
jzxxzjrowrord
@jzxxzjrowrord (jzxxzjrowrord)

Are you sure you want to delete the question?

If your question is resolved, you may close it.

Leaving a resolved question undeleted may help others!

We hope you find it useful!

GCFでpythonを使ってseleniumを使いのですがエラーがでてしまいます

Q&A

PythonSeleniumserverlessGoogleCloud

解決したいこと

GCFでpythonを使ってseleniumを動作させたいのですが、エラーが出て困っています

現在までしてきたこと

1.ローカル環境python3.11でmain.pyを作成しました
2.chromedriver,headless-chromiumをhttps://github.com/ryfeus/gcf-packsにてダウンロード
3.GCFにてファンクションを作成
4.ローカル環境にてmain.py,requirements.tx,chromedriver,headless-chromiumをzipファイルにしてgoogle storeageにアップロード
5.ファンクションをランタイムPython3.11にてデプロイ
6.ファンクションをテスト
7.エラー←いまここ

発生している問題・エラー

長文になりますがGCFコンソールのログを全て記載します(ログでは日時も表示していますが削除しています)
function-1t9jqbeh3hxqv Function execution started 
function-1t9jqbeh3hxqv driver setting 
function-1t9jqbeh3hxqv copy headless-chromium 
function-1t9jqbeh3hxqv copy chromedriver 
function-1t9jqbeh3hxqv get driver 
function-1t9jqbeh3hxqv Memory limit of 256 MiB exceeded with 288 MiB used. Consider increasing the memory limit, see https://cloud.google.com/functions/docs/configuring/memory 
function-1t9jqbeh3hxqv [2023-11-30 12:12:24,913] ERROR in app: Exception on / [POST] 
function-1t9jqbeh3hxqv Traceback (most recent call last):   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/flask/app.py", line 2529, in wsgi_app     response = self.full_dispatch_request() 
function-1t9jqbeh3hxqv                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/flask/app.py", line 1825, in full_dispatch_request 
function-1t9jqbeh3hxqv     rv = self.handle_user_exception(e) 
function-1t9jqbeh3hxqv          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/flask/app.py", line 1823, in full_dispatch_request 
function-1t9jqbeh3hxqv     rv = self.dispatch_request() 
function-1t9jqbeh3hxqv          ^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/flask/app.py", line 1799, in dispatch_request 
function-1t9jqbeh3hxqv     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/functions_framework/__init__.py", line 171, in view_func 
function-1t9jqbeh3hxqv     function(data, context) 
function-1t9jqbeh3hxqv   File "/workspace/main.py", line 63, in main 
function-1t9jqbeh3hxqv     settingDriver() 
function-1t9jqbeh3hxqv   File "/workspace/main.py", line 59, in settingDriver 
function-1t9jqbeh3hxqv     driver = webdriver.Chrome(executable_path=driverPath, options=chrome_options) 
function-1t9jqbeh3hxqv              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/selenium/webdriver/chrome/webdriver.py", line 76, in __init__ 
function-1t9jqbeh3hxqv     RemoteWebDriver.__init__( 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__ 
function-1t9jqbeh3hxqv     self.start_session(capabilities, browser_profile) 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session 
function-1t9jqbeh3hxqv     response = self.execute(Command.NEW_SESSION, parameters) 
function-1t9jqbeh3hxqv                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 319, in execute 
function-1t9jqbeh3hxqv     response = self.command_executor.execute(driver_command, params) 
function-1t9jqbeh3hxqv                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/selenium/webdriver/remote/remote_connection.py", line 374, in execute 
function-1t9jqbeh3hxqv     return self._request(command_info[0], url, body=data) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/selenium/webdriver/remote/remote_connection.py", line 397, in _request 
function-1t9jqbeh3hxqv     resp = self._conn.request(method, url, body=body, headers=headers) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/_request_methods.py", line 118, in request 
function-1t9jqbeh3hxqv     return self.request_encode_body( 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/_request_methods.py", line 217, in request_encode_body 
function-1t9jqbeh3hxqv     return self.urlopen(method, url, **extra_kw) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/poolmanager.py", line 433, in urlopen 
function-1t9jqbeh3hxqv     conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host 
function-1t9jqbeh3hxqv     return self.connection_from_context(request_context) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/poolmanager.py", line 329, in connection_from_context 
function-1t9jqbeh3hxqv     return self.connection_from_pool_key(pool_key, request_context=request_context) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/poolmanager.py", line 352, in connection_from_pool_key 
function-1t9jqbeh3hxqv     pool = self._new_pool(scheme, host, port, request_context=request_context) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/poolmanager.py", line 266, in _new_pool 
function-1t9jqbeh3hxqv     return pool_cls(host, port, **request_context) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/connectionpool.py", line 196, in __init__ 
function-1t9jqbeh3hxqv     timeout = Timeout.from_float(timeout) 
function-1t9jqbeh3hxqv               ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/util/timeout.py", line 190, in from_float 
function-1t9jqbeh3hxqv     return Timeout(read=timeout, connect=timeout) 
function-1t9jqbeh3hxqv            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/util/timeout.py", line 119, in __init__ 
function-1t9jqbeh3hxqv     self._connect = self._validate_timeout(connect, "connect") 
function-1t9jqbeh3hxqv                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
function-1t9jqbeh3hxqv   File "/layers/google.python.pip/pip/lib/python3.11/site-packages/urllib3/util/timeout.py", line 156, in _validate_timeout 
function-1t9jqbeh3hxqv     raise ValueError( 
function-1t9jqbeh3hxqv Function execution took 3614 ms, finished with status: 'crash' 
function-1t9jqbeh3hxqv ValueError: Timeout value connect was <object object at 0x3e44e53848c0>, but it must be an int, float or None.

該当するソースコード

main.py
import os
import shutil
import stat
from pathlib import Path
from selenium import webdriver

global driver


def add_execute_permission(path: Path, target: str = "u"):
    """Add `x` (`execute`) permission to specified targets."""
    mode_map = {
        "u": stat.S_IXUSR,
        "g": stat.S_IXGRP,
        "o": stat.S_IXOTH,
        "a": stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH,
    }

    mode = path.stat().st_mode
    for t in target:
        mode |= mode_map[t]

    path.chmod(mode)


def settingDriver():
    print("driver setting")
    global driver

    driverPath = "/tmp" + "/chromedriver"
    headlessPath = "/tmp" + "/headless-chromium"

    # copy and change permission
    print("copy headless-chromium")
    shutil.copyfile(os.getcwd() + "/headless-chromium", headlessPath)
    add_execute_permission(Path(headlessPath), "ug")

    print("copy chromedriver")
    shutil.copyfile(os.getcwd() + "/chromedriver", driverPath)
    add_execute_permission(Path(driverPath), "ug")

    chrome_options = webdriver.ChromeOptions()

    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--window-size=1280x1696")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--hide-scrollbars")
    chrome_options.add_argument("--enable-logging")
    chrome_options.add_argument("--log-level=0")
    chrome_options.add_argument("--v=99")
    chrome_options.add_argument("--single-process")
    chrome_options.add_argument("--ignore-certificate-errors")
    chrome_options.add_argument("--disable-dev-shm-usage")

    chrome_options.binary_location = headlessPath

    print("get driver")
    driver = webdriver.Chrome(executable_path=driverPath, options=chrome_options)


def main(request1,request2):
    settingDriver()
    global driver
    
    print("URL get")
    
    driver.get("https://en.wikipedia.org/wiki/Special:Random")
    
    print("driver quit")
    
    driver.quit()
requirements.txt
# Function dependencies, for example:
# package>=version
selenium==3.141.0
setuptools
google-cloud-error-reporting==0.30.0
sample.zip
chromedriver
headless-chromium
main.py
requirements.txt

自分で試したこと

・GCFのランタイムをPython3.8､Python3.9,Python3.10にバージョン変更
・ローカル環境にてpip checkにてモジュール間の環境依存関係を確認→No broken requirements foundにて問題なし

0

1Answer

Comments

  1. @jzxxzjrowrord

    Questioner

    コメントありがとうございます
    メモリを512GBに引き上げましたが同じでしたがその部分のエラーログは消えました

Your answer might help someone💌

LoginSign Up