専門用語（キーワード）自動抽出Pythonモジュールtermextract
http://gensen.dl.itc.u-tokyo.ac.jp/pytermextract/

bash

$ docker login
$ docker run -v /tmp/autosar:/tmp/autosar -p 8888:8888 -it continuumio/anaconda3 /bin/bash
Unable to find image 'continuumio/anaconda3:latest' locally
latest: Pulling from continuumio/anaconda3
852e50cd189d: Pull complete 
864e1e8957d3: Pull complete 
6d4823199f64: Pull complete 
Digest: sha256:0b2047cdc438807b87d53272c3d5b1

ubuntu

(base) root@75aa2a1ec996:/# apt update; apt -y upgrade
(base) root@75aa2a1ec996:/# apt install wget vim unzip
(base) root@75aa2a1ec996:/# wget http://gensen.dl.itc.u-tokyo.ac.jp/soft/pytermextract-0_01.zip
(base) root@75aa2a1ec996:/# unzip pytermextract-0_01.zip
Archive: pytermextract-0_01.zip
   creating: pytermextract-0_01/documents/
  inflating: pytermextract-0_01/documents/concept.png  
   creating: pytermextract-0_01/documents/css/
  inflating: pytermextract-0_01/documents/css/pytermextract.css  
   creating: pytermextract-0_01/documents/functions/
  inflating: pytermextract-0_01/documents/functions/chinese_plaintext.html  
  inflating: pytermextract-0_01/documents/functions/english_plaintext.html  
  inflating: pytermextract-0_01/documents/functions/english_postagger.html  
  inflating: pytermextract-0_01/documents/functions/janome.html  
  inflating: pytermextract-0_01/documents/functions/janome_pp.html  
  inflating: pytermextract-0_01/documents/functions/janome_store-lr.html  
  inflating: pytermextract-0_01/documents/functions/janome_tfidf.html  
  inflating: pytermextract-0_01/documents/functions/japanese_plaintext.html  
  inflating: pytermextract-0_01/documents/functions/mecab.html  
  inflating: pytermextract-0_01/documents/functions/nlpir.html  
  inflating: pytermextract-0_01/documents/index.html  
   creating: pytermextract-0_01/pytermex/
  inflating: pytermextract-0_01/pytermex/clear_df.py  
  inflating: pytermextract-0_01/pytermex/clear_store_lr.py  
  inflating: pytermextract-0_01/pytermex/termex_chi_plain.py  
  inflating: pytermextract-0_01/pytermex/termex_eng.py  
  inflating: pytermextract-0_01/pytermex/termex_eng_plain.py  
  inflating: pytermextract-0_01/pytermex/termex_janome.py  
  inflating: pytermextract-0_01/pytermex/termex_janome_pp.py  
  inflating: pytermextract-0_01/pytermex/termex_janome_store_lr1.py  
  inflating: pytermextract-0_01/pytermex/termex_janome_store_lr2.py  
  inflating: pytermextract-0_01/pytermex/termex_janome_tfidf1.py  
  inflating: pytermextract-0_01/pytermex/termex_janome_tfidf2.py  
  inflating: pytermextract-0_01/pytermex/termex_jpn_plain.py  
  inflating: pytermextract-0_01/pytermex/termex_mecab.py  
  inflating: pytermextract-0_01/pytermex/termex_nlpir.py  
 extracting: pytermextract-0_01/readme.txt  
  inflating: pytermextract-0_01/setup.py  
   creating: pytermextract-0_01/termextract/
 extracting: pytermextract-0_01/termextract/__init__.py  
  inflating: pytermextract-0_01/termextract/chinese_plaintext.py  
  inflating: pytermextract-0_01/termextract/core.py  
  inflating: pytermextract-0_01/termextract/english_plaintext.py  
  inflating: pytermextract-0_01/termextract/english_postagger.py  
  inflating: pytermextract-0_01/termextract/janome.py  
  inflating: pytermextract-0_01/termextract/japanese_plaintext.py  
  inflating: pytermextract-0_01/termextract/mecab.py  
  inflating: pytermextract-0_01/termextract/nlpir.py  
   creating: pytermextract-0_01/test_data/
  inflating: pytermextract-0_01/test_data/chi_sample.txt  
  inflating: pytermextract-0_01/test_data/chi_sample_s.txt  
  inflating: pytermextract-0_01/test_data/eng_sample.txt  
  inflating: pytermextract-0_01/test_data/jpn_sample.txt  
  inflating: pytermextract-0_01/test_data/jpn_sample2.txt  
  inflating: pytermextract-0_01/test_data/jpn_sample3.txt  
  inflating: pytermextract-0_01/test_data/mecab_check.txt  
  inflating: pytermextract-0_01/test_data/mecab_out_sample.txt  
   creating: pytermextract-0_01/tests/
 extracting: pytermextract-0_01/tests/__init__.py  
  inflating: pytermextract-0_01/tests/store-lr.bak  
  inflating: pytermextract-0_01/tests/store-lr.dat  
  inflating: pytermextract-0_01/tests/store-lr.dir  
  inflating: pytermextract-0_01/tests/termextrat.bak  
  inflating: pytermextract-0_01/tests/termextrat.dat  
  inflating: pytermextract-0_01/tests/termextrat.dir  
  inflating: pytermextract-0_01/tests/test_chi_plain.py  
  inflating: pytermextract-0_01/tests/test_eng.py  
  inflating: pytermextract-0_01/tests/test_eng_plain.py  
  inflating: pytermextract-0_01/tests/test_janome.py  
  inflating: pytermextract-0_01/tests/test_jpn_plain.py  
  inflating: pytermextract-0_01/tests/test_mecab.py  
  inflating: pytermextract-0_01/tests/test_mecab_pp.py  
  inflating: pytermextract-0_01/tests/test_mecab_store_lr.py  
  inflating: pytermextract-0_01/tests/test_nlpir.py  
(base) root@75aa2a1ec996:/# ls
bin   dev  home  lib64	mnt  p.zip  pytermextract-0_01	run   srv  tmp	var
boot  etc  lib	 media	opt  proc   root		sbin  sys  usr
(base) root@75aa2a1ec996:/# cd pytermextract-0_01/
(base) root@75aa2a1ec996:/pytermextract-0_01# ls
documents  pytermex  readme.txt  setup.py  termextract	test_data  tests
(base) root@75aa2a1ec996:/pytermextract-0_01# python setup.py install
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/termextract
copying termextract/core.py -> build/lib/termextract
copying termextract/nlpir.py -> build/lib/termextract
copying termextract/__init__.py -> build/lib/termextract
copying termextract/english_plaintext.py -> build/lib/termextract
copying termextract/chinese_plaintext.py -> build/lib/termextract
copying termextract/japanese_plaintext.py -> build/lib/termextract
copying termextract/janome.py -> build/lib/termextract
copying termextract/english_postagger.py -> build/lib/termextract
copying termextract/mecab.py -> build/lib/termextract
running install_lib
creating /opt/conda/lib/python3.8/site-packages/termextract
copying build/lib/termextract/core.py -> /opt/conda/lib/python3.8/site-packages/termextract
copying build/lib/termextract/nlpir.py -> /opt/conda/lib/python3.8/site-packages/termextract
copying build/lib/termextract/__init__.py -> /opt/conda/lib/python3.8/site-packages/termextract
copying build/lib/termextract/english_plaintext.py -> /opt/conda/lib/python3.8/site-packages/termextract
copying build/lib/termextract/chinese_plaintext.py -> /opt/conda/lib/python3.8/site-packages/termextract
copying build/lib/termextract/japanese_plaintext.py -> /opt/conda/lib/python3.8/site-packages/termextract
copying build/lib/termextract/janome.py -> /opt/conda/lib/python3.8/site-packages/termextract
copying build/lib/termextract/english_postagger.py -> /opt/conda/lib/python3.8/site-packages/termextract
copying build/lib/termextract/mecab.py -> /opt/conda/lib/python3.8/site-packages/termextract
byte-compiling /opt/conda/lib/python3.8/site-packages/termextract/core.py to core.cpython-38.pyc
byte-compiling /opt/conda/lib/python3.8/site-packages/termextract/nlpir.py to nlpir.cpython-38.pyc
byte-compiling /opt/conda/lib/python3.8/site-packages/termextract/__init__.py to __init__.cpython-38.pyc
byte-compiling /opt/conda/lib/python3.8/site-packages/termextract/english_plaintext.py to english_plaintext.cpython-38.pyc
byte-compiling /opt/conda/lib/python3.8/site-packages/termextract/chinese_plaintext.py to chinese_plaintext.cpython-38.pyc
byte-compiling /opt/conda/lib/python3.8/site-packages/termextract/japanese_plaintext.py to japanese_plaintext.cpython-38.pyc
byte-compiling /opt/conda/lib/python3.8/site-packages/termextract/janome.py to janome.cpython-38.pyc
byte-compiling /opt/conda/lib/python3.8/site-packages/termextract/english_postagger.py to english_postagger.cpython-38.pyc
byte-compiling /opt/conda/lib/python3.8/site-packages/termextract/mecab.py to mecab.cpython-38.pyc
running install_egg_info
Writing /opt/conda/lib/python3.8/site-packages/termextract-0.12b-py3.8.egg-info

(base) root@75aa2a1ec996:/pytermextract-0_01# cd tests
(base) root@75aa2a1ec996:/pytermextract-0_01/tests# python -m unittest
.E.E....E
======================================================================
ERROR: test (test_eng.english_postagger)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/pytermextract-0_01/tests/test_eng.py", line 15, in test
    tagged_text = nltk.pos_tag(nltk.word_tokenize(text))
  File "/opt/conda/lib/python3.8/site-packages/nltk/tokenize/__init__.py", line 129, in word_tokenize
    sentences = [text] if preserve_line else sent_tokenize(text, language)
  File "/opt/conda/lib/python3.8/site-packages/nltk/tokenize/__init__.py", line 106, in sent_tokenize
    tokenizer = load("tokenizers/punkt/{0}.pickle".format(language))
  File "/opt/conda/lib/python3.8/site-packages/nltk/data.py", line 752, in load
    opened_resource = _open(resource_url)
  File "/opt/conda/lib/python3.8/site-packages/nltk/data.py", line 877, in _open
    return find(path_, path + [""]).open()
  File "/opt/conda/lib/python3.8/site-packages/nltk/data.py", line 585, in find
    raise LookupError(resource_not_found)
LookupError: 
**********************************************************************
  Resource punkt not found.
  Please use the NLTK Downloader to obtain the resource:

  >>> import nltk
  >>> nltk.download('punkt')
  
  For more information see: https://www.nltk.org/data.html

  Attempted to load tokenizers/punkt/PY3/english.pickle

  Searched in:
    - '/root/nltk_data'
    - '/opt/conda/nltk_data'
    - '/opt/conda/share/nltk_data'
    - '/opt/conda/lib/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
    - ''
**********************************************************************


======================================================================
ERROR: test (test_janome.janome)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/pytermextract-0_01/tests/test_janome.py", line 9, in test
    from janome.tokenizer import Tokenizer
ModuleNotFoundError: No module named 'janome'

======================================================================
ERROR: test (test_nlpir.nlpir)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/pytermextract-0_01/tests/test_nlpir.py", line 9, in test
    import pynlpir
ModuleNotFoundError: No module named 'pynlpir'

----------------------------------------------------------------------
Ran 9 tests in 6.276s

FAILED (errors=3)

じゃ

ubuntu

# python
Python 3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> nltk.download('punkt')
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
True
>>> nltk.download('averaged_perceptron_tagger')
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
True
>>> exit

# python -m unittest
...E....E
======================================================================
ERROR: test (test_janome.janome)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/pytermextract-0_01/tests/test_janome.py", line 9, in test
    from janome.tokenizer import Tokenizer
ModuleNotFoundError: No module named 'janome'

======================================================================
ERROR: test (test_nlpir.nlpir)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/pytermextract-0_01/tests/test_nlpir.py", line 9, in test
    import pynlpir
ModuleNotFoundError: No module named 'pynlpir'

----------------------------------------------------------------------
Ran 9 tests in 6.164s

FAILED (errors=2)

 pip install janome
Collecting janome
  Downloading Janome-0.4.1-py2.py3-none-any.whl (19.7 MB)
     |████████████████████████████████| 19.7 MB 5.4 MB/s 
Installing collected packages: janome
Successfully installed janome-0.4.1
# python -m unittest
.........
----------------------------------------------------------------------
Ran 9 tests in 7.016s

OK

docker push失敗。

bash

$ docker commit 75aa2a1ec996 kaizenjapan/pytermextract
Error response from daemon: open /var/lib/docker/image/overlay2/layerdb/tmp/write-set-689207880/diff: structure needs cleaning

python - Python:複合語を既知の単語に分割(辞書から)
https://cloud6.net/so/python/2100473

最後までおよみいただきありがとうございました。

いいね　💚、フォローをお願いします。

Thank you very much for reading to the last sentence.

Please press the like icon 💚　and follow me for your happy life.