LoginSignup
1
1

More than 3 years have passed since last update.

@muramasa2

ESPNetのAttributeError: module 'warpctc_pytorch' has no attribute 'gpu_ctc'解決法

Posted at

実行環境

  • Ubuntu18.04(Docker 19.03.12)
  • Python3.7
  • CUDA 10.0

ESPNetのStage4(Network Training)で以下のエラーが発生

dictionary: data/lang_1char/train_nodup_sp_units.txt
stage 4: Network Training
run.pl: job failed, log is in exp/train_nodup_sp_pytorch_train_rnn/train.log
exp/train_nodup_sp_pytorch_train_rnn/train.log
# asr_train.py --config conf/tuning/train_rnn.yaml --ngpu 1 --backend pytorch --outdir exp/train_nodup_sp_pytorch_train_rnn/results --tensorboard-dir tensorboard/train_nodup_sp_pytorch_train_rnn --debugmode 1 --dict data/lang_1char/train_nodup_sp_units.txt --debugdir exp/train_nodup_sp_pytorch_train_rnn --minibatches 0 --seed 1 --verbose 0 --resume --train-json dump/train_nodup_sp/deltafalse/data.json --valid-json dump/train_dev/deltafalse/data.json 
# Started at Mon Jul 20 06:07:11 UTC 2020
#
/espnet/tools/venv/lib/python3.7/site-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.
Import requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.
  from numba.decorators import jit as optional_jit
/espnet/tools/venv/lib/python3.7/site-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.
Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.
  from numba.decorators import jit as optional_jit
2020-07-20 06:07:12,362 (asr_train:555) WARNING: Skip DEBUG/INFO messages
2020-07-20 06:07:12,614 (nets_utils:417) WARNING: Subsampling is not performed for vgg*. It is performed in max pooling layers at CNN.
Exception in main training loop: module 'warpctc_pytorch' has no attribute 'gpu_ctc'
Traceback (most recent call last):
  File "/espnet/tools/venv/lib/python3.7/site-packages/chainer/training/trainer.py", line 316, in run
    update()
  File "/espnet/espnet/asr/pytorch_backend/asr.py", line 242, in update
    self.update_core()
  File "/espnet/espnet/asr/pytorch_backend/asr.py", line 206, in update_core
    data_parallel(self.model, x, range(self.ngpu)).mean() / self.accum_grad
  File "/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 185, in data_parallel
    return module(*inputs[0], **module_kwargs[0])
  File "/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/espnet/espnet/nets/pytorch_backend/e2e_asr.py", line 363, in forward
    self.loss_ctc = self.ctc(hs_pad, hlens, ys_pad)
  File "/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/espnet/espnet/nets/pytorch_backend/ctc.py", line 112, in forward
    self.loss = to_device(self, self.loss_fn(ys_hat, ys_true, hlens, olens)).to(
  File "/espnet/espnet/nets/pytorch_backend/ctc.py", line 62, in loss_fn
    return self.ctc_loss(th_pred, th_target, th_ilen, th_olen)
  File "/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/espnet/tools/venv/lib/python3.7/site-packages/warpctc_pytorch/__init__.py", line 93, in forward
    self.length_average, self.blank, self.reduce)
  File "/espnet/tools/venv/lib/python3.7/site-packages/warpctc_pytorch/__init__.py", line 23, in forward
    loss_func = warp_ctc.gpu_ctc if is_cuda else warp_ctc.cpu_ctc
Will finalize trainer extensions and updater before reraising the exception.
Traceback (most recent call last):
  File "/espnet/egs/csj/asr1/../../../espnet/bin/asr_train.py", line 628, in <module>
    main(sys.argv[1:])
  File "/espnet/egs/csj/asr1/../../../espnet/bin/asr_train.py", line 614, in main
    train(args)
  File "/espnet/espnet/asr/pytorch_backend/asr.py", line 830, in train
    trainer.run()
  File "/espnet/tools/venv/lib/python3.7/site-packages/chainer/training/trainer.py", line 349, in run
    six.reraise(*exc_info)
  File "/espnet/tools/venv/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/espnet/tools/venv/lib/python3.7/site-packages/chainer/training/trainer.py", line 316, in run
    update()
  File "/espnet/espnet/asr/pytorch_backend/asr.py", line 242, in update
    self.update_core()
  File "/espnet/espnet/asr/pytorch_backend/asr.py", line 206, in update_core
    data_parallel(self.model, x, range(self.ngpu)).mean() / self.accum_grad
  File "/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 185, in data_parallel
    return module(*inputs[0], **module_kwargs[0])
  File "/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/espnet/espnet/nets/pytorch_backend/e2e_asr.py", line 363, in forward
    self.loss_ctc = self.ctc(hs_pad, hlens, ys_pad)
  File "/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/espnet/espnet/nets/pytorch_backend/ctc.py", line 112, in forward
    self.loss = to_device(self, self.loss_fn(ys_hat, ys_true, hlens, olens)).to(
  File "/espnet/espnet/nets/pytorch_backend/ctc.py", line 62, in loss_fn
    return self.ctc_loss(th_pred, th_target, th_ilen, th_olen)
  File "/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/espnet/tools/venv/lib/python3.7/site-packages/warpctc_pytorch/__init__.py", line 93, in forward
    self.length_average, self.blank, self.reduce)
  File "/espnet/tools/venv/lib/python3.7/site-packages/warpctc_pytorch/__init__.py", line 23, in forward
    loss_func = warp_ctc.gpu_ctc if is_cuda else warp_ctc.cpu_ctc
AttributeError: module 'warpctc_pytorch' has no attribute 'gpu_ctc'
# Accounting: time=27 threads=1
# Ended (code 1) at Mon Jul 20 06:07:38 UTC 2020, elapsed time 27 seconds

要するにwarpctc_pytorch周りでエラーが起きている

解決法

  1. ESPNetはvenv環境内で実行しているため、まずvenv環境に入る。

    source /espnet/tools/venv/bin/activate

  2. warp ctcの再インストール

git clone https://github.com/espnet/warp-ctc -b pytorch-1.1
cd warp-ctc && mkdir build && cd build && cmake .. && make -j
cd ../pytorch_binding && python(もしくはpython3) setup.py install
  • 以下のエラー出た場合は、3-Aを実行した後、python setup.py installを再実行
running install
running bdist_egg
running egg_info
creating warpctc_pytorch10_cuda100.egg-info
writing warpctc_pytorch10_cuda100.egg-info/PKG-INFO
writing dependency_links to warpctc_pytorch10_cuda100.egg-info/dependency_links.txt
writing top-level names to warpctc_pytorch10_cuda100.egg-info/top_level.txt
writing manifest file 'warpctc_pytorch10_cuda100.egg-info/SOURCES.txt'
reading manifest file 'warpctc_pytorch10_cuda100.egg-info/SOURCES.txt'
writing manifest file 'warpctc_pytorch10_cuda100.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/warpctc_pytorch
copying warpctc_pytorch/__init__.py -> build/lib.linux-x86_64-3.7/warpctc_pytorch
creating build/lib.linux-x86_64-3.7/warpctc_pytorch/lib
copying warpctc_pytorch/lib/libwarpctc.so -> build/lib.linux-x86_64-3.7/warpctc_pytorch/lib
running build_ext
building 'warpctc_pytorch._warp_ctc' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/src
gcc -pthread -B /espnet/tools/venv/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/warp-ctc/include -I/espnet/tools/venv/lib/python3.7/site-packages/torch/lib/include -I/espnet/tools/venv/lib/python3.7/site-packages/torch/lib/include/torch/csrc/api/include -I/espnet/tools/venv/lib/python3.7/site-packages/torch/lib/include/TH -I/espnet/tools/venv/lib/python3.7/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/espnet/tools/venv/include/python3.7m -c src/binding.cpp -o build/temp.linux-x86_64-3.7/src/binding.o -std=c++11 -fPIC -DWARPCTC_ENABLE_GPU -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_warp_ctc -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
src/binding.cpp:11:11: fatal error: c10/cuda/CUDAGuard.h: No such file or directory
  #include "c10/cuda/CUDAGuard.h"
           ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
  • 以下のエラーが出る場合は、3-Bを実行した後、python setup.py installを再実行
running install
running bdist_egg
running egg_info
writing warpctc_pytorch12_cuda100.egg-info/PKG-INFO
writing dependency_links to warpctc_pytorch12_cuda100.egg-info/dependency_links.txt
writing top-level names to warpctc_pytorch12_cuda100.egg-info/top_level.txt
reading manifest file 'warpctc_pytorch12_cuda100.egg-info/SOURCES.txt'
writing manifest file 'warpctc_pytorch12_cuda100.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying warpctc_pytorch/lib/libwarpctc.so -> build/lib.linux-x86_64-3.6/warpctc_pytorch/lib
running build_ext
building 'warpctc_pytorch._warp_ctc' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/warp-ctc/include -I/usr/local/lib/python3.6/dist-packages/torch/include -I/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.6/dist-packages/torch/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c src/binding.cpp -o build/temp.linux-x86_64-3.6/src/binding.o -std=c++11 -fPIC -DWARPCTC_ENABLE_GPU -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_warp_ctc -D_GLIBCXX_USE_CXX11_ABI=0
In file included from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/Device.h:3:0,
                 from /usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/api/include/torch/python.h:8,
                 from /usr/local/lib/python3.6/dist-packages/torch/include/torch/extension.h:6,
                 from src/binding.cpp:6:
/usr/local/lib/python3.6/dist-packages/torch/include/torch/csrc/python_headers.h:9:10: fatal error: Python.h: No such file or directory
 #include <Python.h>
          ^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

3-A. 自分のCUDAバージョンにあったPytorchのインストール

pip install torch===1.2.0 torchvision===0.4.0 -f https://download.pytorch.org/whl/torch_stable.html

3-B. python3.7-devのインストール

apt-get install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt-get update
apt-get install python3.7(自分のpythonのバージョンに合わせる)-dev
1
1
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
1