Airflow on Kubernetesで発生したエラーの内容と対策方法。

目次

【Airflow on Kubernetes】目次

発生バージョン

airflow-1.10.3

エラー内容

scripts/ci/kubernetes/kube/deploy.shを実行しPodを起動しようとすると、initContainersのcommandのairflow initdbの処理で以下のエラーが出る。

+airflow initdb
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 21, in <module>
    from airflow import configuration
  File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 40, in <module>
    from flask_admin import BaseView
  File "/usr/local/lib/python2.7/dist-packages/flask_admin/__init__.py", line 6, in <module>
    from .base import expose, expose_plugview, Admin, BaseView, AdminIndexView  # noqa: F401
  File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 6, in <module>
    from flask import Blueprint, current_app, render_template, abort, g, url_for
  File "/usr/local/lib/python2.7/dist-packages/flask/__init__.py", line 20, in <module>
    from .app import Flask
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 69, in <module>
    from .wrappers import Request
  File "/usr/local/lib/python2.7/dist-packages/flask/wrappers.py", line 14, in <module>
    from werkzeug.wrappers.json import JSONMixin as _JSONMixin
ImportError: No module named json

調査

scripts/ci/kubernetes/docker/Dockerfileをdocker buildすると以下のエラーが出る。

ERROR: flask 1.1.1 has requirement Jinja2>=2.10.1, but you'll have jinja2 2.10 which is incompatible.
ERROR: flask 1.1.1 has requirement Werkzeug>=0.15, but you'll have werkzeug 0.14.1 which is incompatible.
ERROR: pendulum 1.4.4 has requirement tzlocal<2.0.0.0,>=1.5.0.0, but you'll have tzlocal 2.0.0 which is incompatible.

原因

flask 1.1.1では、以下の3つのモジュールが下記のバージョンである必要があるが、1.10.3では古いものが定義されている。

jinja2>=2.10.0
tzlocal>=1.5.0.0, <2.0.0.0
werkzeug>=0.15.0

対策

setup.pyのdef do_setup()の部分を以下のように変更する。

diff --git a/setup.py b/setup.py
index 006de0a..95c40ae 100644
--- a/setup.py
+++ b/setup.py
@@ -307,7 +307,7 @@ def do_setup():
             'gunicorn>=19.5.0, <20.0',
             'iso8601>=0.1.12',
             'json-merge-patch==0.2',
-            'jinja2>=2.7.3, <=2.10.0',
+            'jinja2>=2.10.0',
             'lxml>=4.0.0',
             'markdown>=2.5.2, <3.0',
             'pandas>=0.17.1, <1.0.0',
@@ -324,9 +324,9 @@ def do_setup():
             'text-unidecode==1.2',
             'typing;python_version<"3.5"',
             'thrift>=0.9.2',
-            'tzlocal>=1.4',
+            'tzlocal>=1.5.0.0, <2.0.0.0',
             'unicodecsv>=0.14.1',
-            'werkzeug>=0.14.1, <0.15.0',
+            'werkzeug>=0.15.0',
             'zope.deprecation>=4.0, <5.0',
         ],
         setup_requires=[

参考

Apache Airflow : airflow initdb results in “ImportError: No module named json”