NLU (Natural Language Understanding=自然言語理解) をやってみたくて、
を参考にしながら、dockerで環境を作って、実際に動くところまでやってみる。
Docker環境構築
ホストマシン上でrasa用のディレクトリを作る。
$ mkdir rasa
$ cd rasa
$ vi Dockerfile
Dockerfile に、まずは python が動くimageのみを記述してみる。
FROM python:3.7-slim-stretch
docker-composeで動かしたいので docker-compose.yml を作成
$ vi docker-compose.yml
docker-compose.yml
version: '2'
services:
rasa:
container_name: rasa
build:
context: .
volumes:
- .:/app
buildする
$ docker-compose build
Building rasa
Step 1/1 : FROM python:3.7-slim-stretch
---> c9ec5ac0f580
Successfully built c9ec5ac0f580
Successfully tagged rasa-test_rasa:latest
$
python動くか確認。
$ docker-compose run --rm rasa bash
Creating network "rasa-test_default" with the default driver
root@c02085da1f5f:/# python --version
Python 3.7.7
root@c02085da1f5f:/#
動いた。
Rasaをインストール
を参考にrasa環境を作っていく
root@c02085da1f5f:/# pip install rasa
...(なんかいろいろインストールされる)
Successfully built sanic-jwt absl-py mattermostwrapper colorclass webexteamssdk SQLAlchemy terminaltables future gast termcolor wrapt PyYAML docopt pyrsistent
Failed to build ujson
Installing collected packages: ujson, certifi, six, pycparser, cffi, cryptography, future, python-telegram-bot, tqdm, tabulate, python-crfsuite, sklearn-crfsuite, dnspython, pymongo, ruamel.yaml, numpy, h5py, keras-applications, opt-einsum, grpcio, protobuf, absl-py, google-pasta, gast, termcolor, scipy, wrapt, keras-preprocessing, werkzeug, cachetools, pyasn1, pyasn1-modules, rsa, google-auth, chardet, urllib3, idna, requests, markdown, oauthlib, requests-oauthlib, google-auth-oauthlib, tensorboard, tensorflow-estimator, astor, tensorflow, python-dateutil, PyYAML, docopt, pykwalify, zipp, importlib-metadata, pyrsistent, attrs, jsonschema, greenlet, gevent, redis, PyJWT, sanic-jwt, cloudpickle, kafka-python, rocketchat-API, joblib, scikit-learn, pysocks, pytz, twilio, cycler, pyparsing, kiwisolver, matplotlib, mattermostwrapper, multidict, psycopg2-binary, colorclass, aiofiles, rfc3986, hstspreload, hpack, hyperframe, h2, h11, sniffio, httpx, httptools, uvloop, websockets, sanic, requests-toolbelt, webexteamssdk, httplib2, oauth2client, python-engineio, tzlocal, apscheduler, async-generator, wcwidth, prompt-toolkit, questionary, fbmessenger, sanic-plugins-framework, sanic-cors, humanfriendly, coloredlogs, colorhash, jsonpickle, SQLAlchemy, async-timeout, yarl, aiohttp, pydot, packaging, rasa-sdk, python-socketio, tensorflow-hub, decorator, tensorflow-probability, typeguard, tensorflow-addons, terminaltables, networkx, jmespath, docutils, botocore, s3transfer, boto3, slackclient, pika, rasa
Running setup.py install for ujson ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ingvl92n/ujson/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ingvl92n/ujson/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-46isinhm/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/ujson
cwd: /tmp/pip-install-ingvl92n/ujson/
Complete output (12 lines):
Warning: 'classifiers' should be a list, got type 'filter'
running install
running build
running build_ext
building 'ujson' extension
creating build
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/python
creating build/temp.linux-x86_64-3.7/lib
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I./python -I./lib -I/usr/local/include/python3.7m -c ./python/ujson.c -o build/temp.linux-x86_64-3.7/./python/ujson.o -D_GNU_SOURCE
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ingvl92n/ujson/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ingvl92n/ujson/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-46isinhm/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/ujson Check the logs for full command output.
root@c02085da1f5f:/#
なんかエラー出た。
unable to execute 'gcc': No such file or directory ってことなので、それっぽいのをインストールする
root@c02085da1f5f:/# apt-get update -q -y && apt-get install -q -y build-essential
...(なんかいろいろインストールされる)
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.3) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
root@c02085da1f5f:/#
もっかいrasaのインストールにチャレンジ
root@c02085da1f5f:/# pip install rasa
...(なんかいろいろインストールされる)
Successfully built ujson
Installing collected packages: six, absl-py, cycler, python-dateutil, kiwisolver, numpy, pyparsing, matplotlib, docopt, PyYAML, pykwalify, protobuf, tensorflow-hub, kafka-python, pytz, multidict, pysocks, urllib3, certifi, idna, chardet, requests, PyJWT, twilio, fbmessenger, ruamel.yaml, yarl, attrs, async-timeout, aiohttp, slackclient, terminaltables, aiofiles, httptools, websockets, ujson, h11, hpack, hyperframe, h2, sniffio, rfc3986, hstspreload, httpx, uvloop, sanic, sanic-plugins-framework, sanic-cors, humanfriendly, coloredlogs, rasa-sdk, tzlocal, apscheduler, packaging, typeguard, tensorflow-addons, greenlet, gevent, pika, scipy, google-pasta, astor, termcolor, wrapt, grpcio, pyasn1, pyasn1-modules, rsa, cachetools, google-auth, werkzeug, markdown, oauthlib, requests-oauthlib, google-auth-oauthlib, tensorboard, h5py, keras-applications, tensorflow-estimator, keras-preprocessing, opt-einsum, gast, tensorflow, cloudpickle, decorator, tensorflow-probability, dnspython, pymongo, python-engineio, joblib, scikit-learn, httplib2, oauth2client, SQLAlchemy, requests-toolbelt, future, webexteamssdk, sanic-jwt, wcwidth, prompt-toolkit, colorhash, colorclass, tqdm, rocketchat-API, jmespath, docutils, botocore, s3transfer, boto3, questionary, zipp, importlib-metadata, pyrsistent, jsonschema, mattermostwrapper, jsonpickle, python-socketio, redis, pydot, networkx, pycparser, cffi, cryptography, python-telegram-bot, psycopg2-binary, async-generator, python-crfsuite, tabulate, sklearn-crfsuite, rasa
Successfully installed PyJWT-1.7.1 PyYAML-5.3.1 SQLAlchemy-1.3.15 absl-py-0.9.0 aiofiles-0.4.0 aiohttp-3.6.2 apscheduler-3.6.3 astor-0.8.1 async-generator-1.10 async-timeout-3.0.1 attrs-19.3.0 boto3-1.12.30 botocore-1.15.30 cachetools-4.0.0 certifi-2019.11.28 cffi-1.14.0 chardet-3.0.4 cloudpickle-1.2.2 colorclass-2.2.0 coloredlogs-10.0 colorhash-1.0.2 cryptography-2.8 cycler-0.10.0 decorator-4.4.2 dnspython-1.16.0 docopt-0.6.2 docutils-0.15.2 fbmessenger-6.0.0 future-0.18.2 gast-0.2.2 gevent-1.4.0 google-auth-1.12.0 google-auth-oauthlib-0.4.1 google-pasta-0.2.0 greenlet-0.4.15 grpcio-1.27.2 h11-0.8.1 h2-3.2.0 h5py-2.10.0 hpack-3.0.0 hstspreload-2020.3.25 httplib2-0.17.0 httptools-0.1.1 httpx-0.9.3 humanfriendly-8.1 hyperframe-5.2.0 idna-2.9 importlib-metadata-1.5.2 jmespath-0.9.5 joblib-0.14.1 jsonpickle-1.3 jsonschema-3.2.0 kafka-python-1.4.7 keras-applications-1.0.8 keras-preprocessing-1.1.0 kiwisolver-1.1.0 markdown-3.2.1 matplotlib-3.1.3 mattermostwrapper-2.2 multidict-4.7.5 networkx-2.4 numpy-1.18.2 oauth2client-4.1.3 oauthlib-3.1.0 opt-einsum-3.2.0 packaging-19.0 pika-1.1.0 prompt-toolkit-2.0.10 protobuf-3.11.3 psycopg2-binary-2.8.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pykwalify-1.7.0 pymongo-3.8.0 pyparsing-2.4.6 pyrsistent-0.16.0 pysocks-1.7.1 python-crfsuite-0.9.7 python-dateutil-2.8.1 python-engineio-3.11.2 python-socketio-4.4.0 python-telegram-bot-11.1.0 pytz-2019.3 questionary-1.5.1 rasa-1.9.2 rasa-sdk-1.9.0 redis-3.4.1 requests-2.23.0 requests-oauthlib-1.3.0 requests-toolbelt-0.9.1 rfc3986-1.3.2 rocketchat-API-0.6.36 rsa-4.0 ruamel.yaml-0.15.100 s3transfer-0.3.3 sanic-19.12.2 sanic-cors-0.10.0.post3 sanic-jwt-1.3.2 sanic-plugins-framework-0.9.2 scikit-learn-0.22.2.post1 scipy-1.4.1 six-1.14.0 sklearn-crfsuite-0.3.6 slackclient-2.5.0 sniffio-1.1.0 tabulate-0.8.7 tensorboard-2.1.1 tensorflow-2.1.0 tensorflow-addons-0.8.3 tensorflow-estimator-2.1.0 tensorflow-hub-0.7.0 tensorflow-probability-0.7.0 termcolor-1.1.0 terminaltables-3.1.0 tqdm-4.31.1 twilio-6.26.3 typeguard-2.7.1 tzlocal-2.0.0 ujson-1.35 urllib3-1.25.8 uvloop-0.14.0 wcwidth-0.1.9 webexteamssdk-1.1.1 websockets-8.1 werkzeug-1.0.0 wrapt-1.12.1 yarl-1.4.2 zipp-3.1.0
root@c02085da1f5f:/#
インストールできた。
次はrasaのinit。
の前に、作業ディレクトリを移動する。/ 直下でなんかファイルができると気持ち悪いので。
root@c02085da1f5f:/# pwd
/
root@c02085da1f5f:/# ls -al
total 88
drwxr-xr-x 1 root root 4096 Mar 27 08:25 .
drwxr-xr-x 1 root root 4096 Mar 27 08:25 ..
-rwxr-xr-x 1 root root 0 Mar 27 08:25 .dockerenv
drwxr-xr-x 4 root root 128 Mar 27 08:22 app
drwxr-xr-x 1 root root 4096 Mar 27 08:33 bin
drwxr-xr-x 2 root root 4096 Feb 1 17:09 boot
drwxr-xr-x 5 root root 360 Mar 27 08:25 dev
drwxr-xr-x 1 root root 4096 Mar 27 08:33 etc
drwxr-xr-x 2 root root 4096 Feb 1 17:09 home
drwxr-xr-x 1 root root 4096 Mar 27 08:33 lib
drwxr-xr-x 2 root root 4096 Feb 24 00:00 lib64
drwxr-xr-x 2 root root 4096 Feb 24 00:00 media
drwxr-xr-x 2 root root 4096 Feb 24 00:00 mnt
drwxr-xr-x 2 root root 4096 Feb 24 00:00 opt
dr-xr-xr-x 228 root root 0 Mar 27 08:25 proc
drwx------ 1 root root 4096 Mar 27 08:27 root
drwxr-xr-x 3 root root 4096 Feb 24 00:00 run
drwxr-xr-x 2 root root 4096 Feb 24 00:00 sbin
drwxr-xr-x 2 root root 4096 Feb 24 00:00 srv
dr-xr-xr-x 13 root root 0 Mar 27 08:25 sys
drwxrwxrwt 1 root root 4096 Mar 27 08:36 tmp
drwxr-xr-x 1 root root 4096 Feb 24 00:00 usr
drwxr-xr-x 1 root root 4096 Feb 24 00:00 var
root@c02085da1f5f:/# cd app/
root@c02085da1f5f:/app# ls -al
total 12
drwxr-xr-x 4 root root 128 Mar 27 08:22 .
drwxr-xr-x 1 root root 4096 Mar 27 08:25 ..
-rw-r--r-- 1 root root 29 Mar 27 08:20 Dockerfile
-rw-r--r-- 1 root root 112 Mar 27 08:22 docker-compose.yml
root@c02085da1f5f:/app#
/app で作業する。
rasa init する
root@c02085da1f5f:/app# rasa init --no-prompt
Welcome to Rasa! 🤖
To get started quickly, an initial project will be created.
If you need some help, check out the documentation at https://rasa.com/docs/rasa.
Created project directory at '/app'.
Finished creating project structure.
Training an initial model...
Training Core model...
(なんやかんや)
2020-03-27 08:46:25 INFO rasa.nlu.model - Successfully saved model into '/tmp/tmp4p2k58s6/nlu'
NLU model training completed.
Your Rasa model is trained and saved at '/app/models/20200327-084527.tar.gz'.
If you want to speak to the assistant, run 'rasa shell' at any time inside the project directory.
root@c02085da1f5f:/app#
init できたっぽい。
ディレクトリ内を見てみる。
root@c02085da1f5f:/app# ls -al
total 32
drwxr-xr-x 14 root root 448 Mar 27 08:46 .
drwxr-xr-x 1 root root 4096 Mar 27 08:25 ..
-rw-r--r-- 1 root root 29 Mar 27 08:20 Dockerfile
-rw-r--r-- 1 root root 0 Mar 27 08:36 __init__.py
drwxr-xr-x 4 root root 128 Mar 27 08:45 __pycache__
-rw-r--r-- 1 root root 757 Mar 27 08:36 actions.py
-rw-r--r-- 1 root root 622 Mar 27 08:36 config.yml
-rw-r--r-- 1 root root 938 Mar 27 08:36 credentials.yml
drwxr-xr-x 4 root root 128 Mar 27 08:45 data
-rw-r--r-- 1 root root 112 Mar 27 08:22 docker-compose.yml
-rw-r--r-- 1 root root 549 Mar 27 08:36 domain.yml
-rw-r--r-- 1 root root 1456 Mar 27 08:36 endpoints.yml
drwxr-xr-x 3 root root 96 Mar 27 08:46 models
drwxr-xr-x 3 root root 96 Mar 27 08:45 tests
root@c02085da1f5f:/app#
なんか色々増えてる。
参考にした記事と同様に、まずは会話してみる。
root@c02085da1f5f:/app# rasa shell
2020-03-27 08:54:07 INFO root - Connecting to channel 'cmdline' which was specified by the '--connector' argument. Any other channels will be ignored. To connect to all given channels, omit the '--connector' argument.
2020-03-27 08:54:07 INFO root - Starting Rasa server on http://localhost:5005
2020-03-27 08:54:08.593814: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
Bot loaded. Type a message and press enter (use '/stop' to exit):
Your input -> hello
Hey! How are you?
Your input -> /stop
2020-03-27 08:54:31 INFO root - Killing Sanic server now.
root@c02085da1f5f:/app#
おお。できてる。
日本語に対応する
次にこの記事に戻って日本語対応してみる。
まずは data/nlu.md の一番下にデータを追加してみる。
$ vi data/nlu.md
data/nlu.md
## intent:restaurant_ja
- 渋谷で美味しいイタリアンない？
- 和食食べたいんだけど、六本木におすすめある？
- 今度麻布行くんだけど、フレンチのお店教えて
上記記事だとmecabをやめてspaCyを使っているようなので、mecabを飛ばしてspaCyでやってみる。
config.yml にデフォルト設定があるようなので、一旦全部消して以下を書いてみる。
config.yml
pipeline:
# - name: “tokenizer_whitespace”
- name: “nlp_spacy”
- name: “tokenizer_spacy”
- name: “CRFEntityExtractor”
- name: “ner_crf”
- name: “ner_synonyms”
- name: “intent_featurizer_count_vectors”
- name: “intent_classifier_tensorflow_embedding”
学習させてみる。
root@c02085da1f5f:/app# rasa train nlu
The config file 'config.yml' is missing mandatory parameters: 'language'. Add missing parameters to config file and try again.
root@c02085da1f5f:/app#
config.yml に languageが無いよって。
上記記事だとja_ginzaにしているようなので、端折って config.yml の一番上に ja_ginza を追加してみる。
config.yml
language: ja_ginza
もういっかいコマンドを叩いてみる。
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/rasa/nlu/registry.py", line 173, in get_component_class
return class_from_module_path(component_name)
File "/usr/local/lib/python3.7/site-packages/rasa/utils/common.py", line 211, in class_from_module_path
raise ImportError(f"Cannot retrieve class from path {module_path}.")
ImportError: Cannot retrieve class from path “nlp_spacy”.
(なんやかんや)
nlp_spacyが云々で怒られている。
spaCy入れてないもんね。
spaCy入れてみる。
root@c02085da1f5f:/app# pip install spacy
(なんやかんや)
Successfully installed blis-0.4.1 catalogue-1.0.0 cymem-2.0.3 murmurhash-1.0.2 plac-1.1.3 preshed-3.0.2 spacy-2.2.4 srsly-1.0.2 thinc-7.4.0 tqdm-4.43.0 wasabi-0.6.0
root@c02085da1f5f:/app#
入った。
GiNZAの日本語処理ライブラリ？があるようなので、インストールする。
root@c02085da1f5f:/app# pip install "https://github.com/megagonlabs/ginza/releases/download/latest/ginza-latest.tar.gz"
Collecting https://github.com/megagonlabs/ginza/releases/download/latest/ginza-latest.tar.gz
(なんやかんや)
Successfully built ginza ja-ginza SudachiDict-core
Installing collected packages: ja-ginza, sortedcontainers, Cython, dartsclone, SudachiPy, SudachiDict-core, ginza
Successfully installed Cython-0.29.16 SudachiDict-core-20190927 SudachiPy-0.4.3 dartsclone-0.9.0 ginza-2.2.1 ja-ginza-2.2.0 sortedcontainers-2.1.0
root@c02085da1f5f:/app#
インストールできた。
これで準備できたか？
もう一回コマンドを叩く。
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/rasa/nlu/registry.py", line 173, in get_component_class
return class_from_module_path(component_name)
File "/usr/local/lib/python3.7/site-packages/rasa/utils/common.py", line 211, in class_from_module_path
raise ImportError(f"Cannot retrieve class from path {module_path}.")
ImportError: Cannot retrieve class from path “nlp_spacy”.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/rasa", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/rasa/__main__.py", line 91, in main
cmdline_arguments.func(cmdline_arguments)
File "/usr/local/lib/python3.7/site-packages/rasa/cli/train.py", line 140, in train_nlu
persist_nlu_training_data=args.persist_nlu_data,
File "/usr/local/lib/python3.7/site-packages/rasa/train.py", line 414, in train_nlu
persist_nlu_training_data,
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.7/site-packages/rasa/train.py", line 445, in _train_nlu_async
persist_nlu_training_data=persist_nlu_training_data,
File "/usr/local/lib/python3.7/site-packages/rasa/train.py", line 474, in _train_nlu_with_validated_data
persist_nlu_training_data=persist_nlu_training_data,
File "/usr/local/lib/python3.7/site-packages/rasa/nlu/train.py", line 74, in train
trainer = Trainer(nlu_config, component_builder)
File "/usr/local/lib/python3.7/site-packages/rasa/nlu/model.py", line 142, in __init__
components.validate_requirements(cfg.component_names)
File "/usr/local/lib/python3.7/site-packages/rasa/nlu/components.py", line 51, in validate_requirements
component_class = registry.get_component_class(component_name)
File "/usr/local/lib/python3.7/site-packages/rasa/nlu/registry.py", line 199, in get_component_class
raise ModuleNotFoundError(exception_message)
ModuleNotFoundError: Cannot find class '“nlp_spacy”' from global namespace. Please check that there is no typo in the class name and that you have imported the class into the global namespace.
root@c02085da1f5f:/app#
なんか怒られた。
Cannot find class '“nlp_spacy”' ってなんだろ。
ModuleNotFoundError: Cannot find class '“nlp_spacy”' from global namespace. Please check that there is no typo in the class name and that you have imported the class into the global namespace. でググってみる。
このページが一番上に出てきた。
old_style_names ？
"nlp_spacy": "SpacyNLP",
こんな感じの対応付けが定義されている。
さっきconfig.ymlに書いたのが、nlp_spacy だから SpacyNLP に変更してみる。
config.yml
language: ja_ginza
pipeline:
# - name: “tokenizer_whitespace”
- name: "SpacyNLP" # ←ここを変更
- name: "tokenizer_spacy"
- name: "CRFEntityExtractor"
- name: "ner_crf"
- name: "ner_synonyms"
- name: "intent_featurizer_count_vectors"
- name: "intent_classifier_tensorflow_embedding"
再度コマンドを叩く。
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...
(なんやかんや)
ModuleNotFoundError: Cannot find class '“SpacyNLP”' from global namespace. Please check that there is no typo in the class name and that you have imported the class into the global namespace.
“SpacyNLP” ？？
あ、コピペしたらダブルクオートがおかしいのか。
ダブルクオートを治す。
config.yml
language: ja_ginza
pipeline:
# - name: “tokenizer_whitespace”
- name: "nlp_spacy"
- name: "tokenizer_spacy"
- name: "CRFEntityExtractor"
- name: "ner_crf"
- name: "ner_synonyms"
- name: "intent_featurizer_count_vectors"
- name: "intent_classifier_tensorflow_embedding"
もういっかいコマンド叩く。
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...
(なんやかんや)
oblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
warnings.warn(msg, category=FutureWarning)
2020-03-27 09:15:20 INFO rasa.nlu.model - Successfully saved model into '/tmp/tmpn4xtb3h7/nlu'
NLU model training completed.
Your Rasa model is trained and saved at '/app/models/nlu-20200327-091520.tar.gz'.
root@c02085da1f5f:/app#
お、やっと動いた。
rasa shell nlu で intent が取れれば成功？
root@c02085da1f5f:/app# rasa shell nlu
2020-03-27 09:16:26 INFO rasa.nlu.components - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-ja_ginza'.
/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
warnings.warn(msg, category=FutureWarning)
2020-03-27 09:16:26.359929: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
/usr/local/lib/python3.7/site-packages/rasa/nlu/classifiers/diet_classifier.py:864: FutureWarning: 'EmbeddingIntentClassifier' is deprecated and will be removed in version 2.0. Use 'DIETClassifier' instead.
model=model,
NLU model loaded. Type a message and press enter to parse it.
Next message:
六本木のイタリアン教えて
{
"intent": {
"name": "restaurant_ja",
"confidence": 0.48260971903800964
},
"entities": [],
"intent_ranking": [
{
"name": "restaurant_ja",
"confidence": 0.48260971903800964
},
{
"name": "affirm",
"confidence": 0.1126396581530571
},
{
"name": "greet",
"confidence": 0.09804855287075043
},
{
"name": "goodbye",
"confidence": 0.08541827648878098
},
{
"name": "mood_great",
"confidence": 0.07586738467216492
},
{
"name": "deny",
"confidence": 0.06578702479600906
},
{
"name": "bot_challenge",
"confidence": 0.05436018109321594
},
{
"name": "mood_unhappy",
"confidence": 0.025269268080592155
}
],
"text": "六本木のイタリアン教えて"
}
Next message:
お、それっぽい。intentのスコア低いけど、こんなもん？
固有表現もやってみよう。
data/nlu.md の日本語の箇所を下記のように書き換えて、
data/nlu.md
## intent:restaurant_ja
- [渋谷](location)で美味しい[イタリアン](restaurant_type)ない？
- [和食](restaurant_type)食べたいんだけど、[六本木](location)におすすめある？
- 今度[麻布](location)行くんだけど、[フレンチ](restaurant_type)のお店教えて
学習させて、
root@c02085da1f5f:/app# rasa train nlu
Training NLU model...
(なんやかんや)
2020-03-27 13:46:37 INFO rasa.nlu.model - Finished training component.
/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
warnings.warn(msg, category=FutureWarning)
2020-03-27 13:46:37 INFO rasa.nlu.model - Successfully saved model into '/tmp/tmpy7tqyrsb/nlu'
NLU model training completed.
Your Rasa model is trained and saved at '/app/models/nlu-20200327-134637.tar.gz'.
root@c02085da1f5f:/app#
shell を叩いてみる。
root@c02085da1f5f:/app# rasa shell nlu
2020-03-27 13:47:07 INFO rasa.nlu.components - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-ja_ginza'.
/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
warnings.warn(msg, category=FutureWarning)
2020-03-27 13:47:07.924760: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
/usr/local/lib/python3.7/site-packages/rasa/nlu/classifiers/diet_classifier.py:864: FutureWarning: 'EmbeddingIntentClassifier' is deprecated and will be removed in version 2.0. Use 'DIETClassifier' instead.
model=model,
NLU model loaded. Type a message and press enter to parse it.
Next message:
渋谷で美味しいイタリアンない？
{
"intent": {
"name": "restaurant_ja",
"confidence": 0.9823814034461975
},
"entities": [
{
"start": 0,
"end": 2,
"value": "渋谷",
"entity": "location",
"confidence": 0.7429793877231764,
"extractor": "CRFEntityExtractor"
},
{
"start": 7,
"end": 12,
"value": "イタリアン",
"entity": "restaurant_type",
"confidence": 0.7588623133724827,
"extractor": "CRFEntityExtractor"
},
{
"start": 0,
"end": 2,
"value": "渋谷",
"entity": "location",
"confidence": 0.7429793877231764,
"extractor": "CRFEntityExtractor"
},
{
"start": 7,
"end": 12,
"value": "イタリアン",
"entity": "restaurant_type",
"confidence": 0.7588623133724827,
"extractor": "CRFEntityExtractor"
}
],
"intent_ranking": [
{
"name": "restaurant_ja",
"confidence": 0.9823814034461975
},
{
"name": "mood_great",
"confidence": 0.008284788578748703
},
{
"name": "deny",
"confidence": 0.006644146051257849
},
{
"name": "greet",
"confidence": 0.001571052591316402
},
{
"name": "bot_challenge",
"confidence": 0.0006475948612205684
},
{
"name": "affirm",
"confidence": 0.00034828390926122665
},
{
"name": "mood_unhappy",
"confidence": 8.46567636472173e-05
},
{
"name": "goodbye",
"confidence": 3.805106462095864e-05
}
],
"text": "渋谷で美味しいイタリアンない？"
}
Next message:
おお。記事みたいな出力になった。
記事とは違って日本語は文字化けしてないけど。
環境ができて動くところまで見れたので、今回はここまで。
今後は教師データを増やす方法を考える。
追記
ここまでの成果として、pip で install したライブラリを requirements.txt にして、build したら勝手にインストールされるように Dockerfile に処理を追記しておいた。
requirements.txt
rasa==1.9.2
spacy==2.2.4
FROM python:3.7-slim-stretch
RUN apt-get update -q -y && \
apt-get install -q -y \
build-essential
ADD ./requirements.txt requirements.txt
RUN pip install -r requirements.txt
RUN pip install "https://github.com/megagonlabs/ginza/releases/download/latest/ginza-latest.tar.gz"
ADD . /app
WORKDIR /app
んで、build して
$ docker-compose build
動かす
$ docker-compose run --rm rasa bash
root@56ff2c2f2f38:/app#
追記2
github で公開した
追記3
ginzaのインストールはpip経由で(3.0から)できるようになったようなので、requirements.txtにginzaを追記してDockerfileから該当箇所を削除した。