Go to Qiita Advent Calendar 2024 Top
LoginSignup
2
3

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@vye03702(VIN)

Jupyter Notebookで訓練したモデルをAzure App Service上で動かしてみる

Last updated at Posted at 2019-05-18

ずいぶん昔の話になりますが、卒論の音声認識では声道の特性を表している線形予測モデルを用いて識別していました。(30数年前の理論が実装されている訳ですから、ITの世界は早いといっても、基本は変わっていませんね！)

今の機械学習では、モデルが識別しようとするものを表しているのかは明確ではなく、非常に多くのパラメータのモデルを、膨大な実例解を力技で解いて、その結果が使えるのなら良しとしており、ちょっと乱暴かなと思っています。

せめて得られたモデルを可視化したり、複数のモデルの比較する事が重要で、Jupyter Notebookでいろいろやる訳ですが、そうやって得たモデルをAzure App Service上で動かしてみました。

###demo
※07/02 15:00 Flask環境からDjango環境へ変更しました。おいおい変更点やソースの情報を掲載します。
#手順
今回のFLASKサーバはpython拡張モジュールを入れて、ライブラリを追加する形を取りました。

1.Jupyter Notebookでモデルを作成し、pickleを用いてsavファイルを吐き出す。

2.AzureのWebappsにPythonサーバを作成。

3.webappsダッシュボードの「開発ツール」→「拡張機能」→「追加」でPython 3.6.4 x64を追加。

※注意：モデルを作成する環境とpythonのバージョン(3.6*レベル)とbit数は合せて下さい。Scikits-Learn RandomForrest trained on 64bit python wont open on 32bit pythonにあるようにbit数が合わないとロード時にエラーになり、私はだいぶロスりました。

4.webappsダッシュボードの「コンソール」 からチェンジディレクトリーで/home/python364x64に移動し、ライブラリを追加。

python -m pip install --upgrade pip

pip install scikit-learn
「 WARNING: The script f2py.exe is installed ・・」と表示されますが、インストールは完了しています。

pip install Flask

pip install pandas

pip install matplotlib

pickleは標準ライブラリに含まれているのでインストールする必要はありません。

5.Jupyter Notebookで作成したアプリケーションをベースにWebAppsアプリを作成する。

6.1で作成したsavファイルをftpで所定の位置にアップする。

といった手順で行いました。

#Jupyter Notebookでモデルを作成でモデルを作成

Azure ML Studioこれについては、タイタニックの生存者データを分析してみたを参考に、データはAzureMLの「米国国勢調査局提供の、成人収入に関する二項分類データセット」を用いて作成しました。

tree可視化
########################################################
#  
#  参考：http://www.randpy.tokyo/entry/python_random_forest
#        
########################################################
#!pip install pydotplus

import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv('Data/Adult Census Income Binary Classification dataset.csv')

#print(df.head(50))

try:
    df['workclass'] = df['workclass'].map( {
    '?': 99,
    'Federal-gov':0,
    'Local-gov':1,
    'Never-worked':2,
    'Private':3,
    'Self-emp-inc':4,
    'Self-emp-not-inc':5,
    'State-gov':6,
    'Without-pay':7,
    } ).astype(int)
except:
    print('erro1')

try:
    df['occupation'] = df['occupation'].map( {
    '?': 99,
    'Adm-clerical':0,
    'Armed-Forces':1,
    'Craft-repair':2,
    'Exec-managerial':3,
    'Farming-fishing':4,
    'Handlers-cleaners':5,
    'Machine-op-inspct':6,
    'Other-service':7,
    'Priv-house-serv':8,
    'Prof-specialty':9,
    'Protective-serv':10,
    'Sales':11,
    'Tech-support':12,
    'Transport-moving':13,      
    } ).astype(int)
except:
    print('erro2')  
    
try:
    df['race'] = df['race'].map( {
    'White':0,
    'Amer-Indian-Eskimo':1,
    'Asian-Pac-Islander':2,
    'Black':4,
    'Other':5,
    } ).astype(int)
except:
    print('erro3')  
    
try:
    df['sex'] = df['sex'].map( {
        'Male': 0,
        'Female': 1,
        'Other': 2
    } ).astype(int)
except:
    print('erro4')  
    
try:
    df['native-country'] = df['native-country'].map( {
        '?': 99,
        'Cambodia':0,
        'Canada':1,
        'China':2,
        'Columbia':3,
        'Cuba':4,
        'Dominican-Republic':5,
        'Ecuador':6,
        'El-Salvador':7,
        'England':8,
        'France':9,
        'Germany':10,
        'Greece':11,
        'Guatemala':12,
        'Haiti':13,
        'Holand-Netherlands':14,
        'Honduras':15,
        'Hong':16,
        'Hungary':17,
        'India':18,
        'Iran':19,
        'Ireland':20,
        'Italy':21,
        'Jamaica':22,
        'Japan':23,
        'Laos':24,
        'Mexico':25,
        'Nicaragua':26,
        'Outlying-US(Guam-USVI-etc)':27,
        'Peru':28,
        'Philippines':29,
        'Poland':30,
        'Portugal':31,
        'Puerto-Rico':32,
        'Scotland':33,
        'South':34,
        'Taiwan':35,
        'Thailand':36,
        'Trinadad&Tobago':37,
        'United-States':38,
        'Vietnam':39,
        'Yugoslavia':40,
    } ).astype(int)
except:
    print('erro5')  

try:
    df['income'] = df['income'].map( {
        '<=50K': 0,
        '>50K': 1,
    } ).astype(int)
except:
    print('erro6')      

df = df.drop(['fnlwgt','education','marital-status','relationship'],axis=1)

train_x = df.drop('income', axis=1)
train_y = df.income
(train_x, test_x ,train_y, test_y) = train_test_split(train_x, train_y, test_size = 0.1, random_state = 666)
    
#決定木
from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier(random_state=0)
clf = clf.fit(train_x, train_y)
pred = clf.predict(test_x)
print('pred = ' , pred)

#評価
from sklearn.metrics import (roc_curve, auc, accuracy_score)

fpr, tpr, thresholds = roc_curve(test_y, pred, pos_label=1)
auc(fpr, tpr)
print('accuracy_score = ' , accuracy_score(pred, test_y))

#可視化
from sklearn import tree
from sklearn.externals.six import StringIO

import pydotplus
from graphviz import Digraph
from PIL import Image

import matplotlib.pyplot as plt
import numpy as np

dot_data = StringIO()

tree.export_graphviz(clf, out_file=dot_data,feature_names=train_x.columns, max_depth=3)

graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
graph.write_jpeg("graph.jpeg")

im = Image.open("graph.jpeg")
plt.figure(figsize=(10, 10), dpi=500)
plt.imshow(im)

#モデルの保存
import pickle
filename = 'Data/finalized_model.sav'
pickle.dump(clf, open(filename, 'wb'))

print('-----Save End-----')

#Jupyter Notebookで判断プログラムを作成しWebappsに処理部分をmain.pyに作成

Jupyter Notebookのプログラムは、省きます。入力のところがcsvのインプットファイルを使っているのと、最後returnで戻すところがprint文なところが違うぐらいです。

main.py
import os
import sys
import platform

from flask import *

import json

import sqlite3

import werkzeug
import datetime

import time

import urllib.request

from sklearn import datasets
from sklearn import svm
from sklearn.model_selection import train_test_split

import pickle
import pandas as pd
 
#**************************************
# ログ出力  if os.path.exists('err.log'):  else:
#**************************************
def log(msg):
    errlog = open('./LogFiles/err.log','a')
    errlog.write( msg + '\n')
    errlog.close()
#**************************************
# 初期メニュー
#**************************************
application = Flask(__name__)
application.config.from_object(__name__)

@application.route('/')
def index():

    return render_template('index.html')
        
#**************************************
# Adult Census Income Binary Classification dataset
#**************************************
#html表示
@application.route('/Income', methods=['POST'])
def Income():
    return render_template('Income.html')

#ajax
@application.route('/IncomeCal', methods=['POST'])
def IncomeCal():

    df = pd.io.json.json_normalize(request.json)

    try:
        df['workclass'] = df['workclass'].map( {
        '?': 99,
        'Federal-gov':0,
        'Local-gov':1,
        'Never-worked':2,
        'Private':3,
        'Self-emp-inc':4,
        'Self-emp-not-inc':5,
        'State-gov':6,
        'Without-pay':7,
        } ).astype(int)
    except:
        log('erro1')
    
    try:
        df['occupation'] = df['occupation'].map( {
        '?': 99,
        'Adm-clerical':0,
        'Armed-Forces':1,
        'Craft-repair':2,
        'Exec-managerial':3,
        'Farming-fishing':4,
        'Handlers-cleaners':5,
        'Machine-op-inspct':6,
        'Other-service':7,
        'Priv-house-serv':8,
        'Prof-specialty':9,
        'Protective-serv':10,
        'Sales':11,
        'Tech-support':12,
        'Transport-moving':13,      
        } ).astype(int)
    except:
        log('erro2')  
        
    try:
        df['race'] = df['race'].map( {
        'White':0,
        'Amer-Indian-Eskimo':1,
        'Asian-Pac-Islander':2,
        'Black':4,
        'Other':5,
        } ).astype(int)
    except:
        log('erro3')  
        
    try:
        df['sex'] = df['sex'].map( {
            'Male': 0,
            'Female': 1,
            'Other': 2
        } ).astype(int)
    except:
        log('erro4')  
        
    try:
        df['native-country'] = df['native-country'].map( {
            '?': 99,
            'Cambodia':0,
            'Canada':1,
            'China':2,
            'Columbia':3,
            'Cuba':4,
            'Dominican-Republic':5,
            'Ecuador':6,
            'El-Salvador':7,
            'England':8,
            'France':9,
            'Germany':10,
            'Greece':11,
            'Guatemala':12,
            'Haiti':13,
            'Holand-Netherlands':14,
            'Honduras':15,
            'Hong':16,
            'Hungary':17,
            'India':18,
            'Iran':19,
            'Ireland':20,
            'Italy':21,
            'Jamaica':22,
            'Japan':23,
            'Laos':24,
            'Mexico':25,
            'Nicaragua':26,
            'Outlying-US(Guam-USVI-etc)':27,
            'Peru':28,
            'Philippines':29,
            'Poland':30,
            'Portugal':31,
            'Puerto-Rico':32,
            'Scotland':33,
            'South':34,
            'Taiwan':35,
            'Thailand':36,
            'Trinadad&Tobago':37,
            'United-States':38,
            'Vietnam':39,
            'Yugoslavia':40,
        } ).astype(int)
    except:
        log('erro5')  
    
    train_x = df
       
    # 保存したモデルをロードする
    filename = './Data/finalized_model.sav'    
    loaded_model = pickle.load(open(filename, 'rb'))
    result = loaded_model.predict(train_x)

    if result[0] == 0:
        result = "<=50"
    else:
        result = ">50"
    return result

#************************************** 
#  サーバ起動  マルチスレッド指定 デフォルトはTrueの動きをするようだが。 https://qiita.com/5zm/items/251be97d2800bf67b1c6
#************************************** 
if __name__ == '__main__':
    application.debug = True # デバッグ
    application.run(host='0.0.0.0', port=8000, threaded=True)
クライアントソースIncome.html
<!--*****************************************************************
*  azureMLサンプル:米国国勢調査提供の収入のサンプルの訓練結果にrestで接続
*  
*  2019/03/24
*****************************************************************-->
{% extends "base.html" %}
{% block body %}


<div class="card" >
<div class="card-header" style="height:50px; font-size:1.5rem; ">
    <b>azure ML webアプリサンプル</b>
</div>
<div class="card-body">
	<div class="row">
		<div class="col-md-2 mb-1">年齢</div> 
		<div class="col-md-2 mb-1" >
			<select class="form-control input-lg mb-1" id="age">
				<option>20</option>
				<option>25</option>
				<option>30</option>
				<option>35</option>
				<option>40</option>
				<option>45</option>				
				<option>50</option>
				<option>55</option>				
				<option>60</option>
				<option>65</option>				
				<option>70</option>
				<option>75</option>				
				<option>80</option>
			</select>
		</div>
		<div class="col-md-2 mb-1">ワーククラス</div>
		<div class="col-md-2 mb-1" >
			<select class="form-control input-lg mb-1" id="workclass">
				<option value="Federal-gov">Federal-gov</option>
				<option value="Local-gov">Local-gov</option>
				<option value="Never-worked">Never-worked</option>
				<option value="Private">Private</option>
				<option value="Self-emp-inc">Self-emp-inc</option>
				<option value="Self-emp-not-inc">Self-emp-not-inc</option>
				<option value="State-gov">State-gov</option>
				<option value="Without-pay">Without-pay</option>
				<option value="Without-pay">Without-pay</option>
			</select>
		</div>
		
		<div class="col-md-1 mb-1">学校</div>
		<div class="col-md-3 mb-1" >
			<select class="form-control input-lg mb-1" id="education-num">
				<option value="1">Preschool</option>
				<option value="2">1st-4th</option>
				<option value="3">5th-6th</option>
				<option value="4">7th-8th</option>
				<option value="5">9th</option>
				<option value="6">10th</option>
				<option value="7">11th</option>
				<option value="8">12th</option>
				<option value="9">HS-grad</option>
				<option value="10">Some-college</option>
				<option value="11">Assoc-voc</option>
				<option value="12">Assoc-acdm</option>
				<option value="13">Bachelors</option>
				<option value="14">Masters</option>
				<option value="15">Prof-school</option>
				<option value="16">Doctorate</option>
			</select>
		</div>
		
	</div>
	  
	<div class="row">

		<div class="col-md-1 mb-1">職業</div>
		<div class="col-md-3 mb-1" >
			<select class="form-control input-lg mb-1" id="occupation">
				<option value="Adm-clerical">Adm-clerical</option>
				<option value="Armed-Forces">Armed-Forces</option>
				<option value="Craft-repair">Craft-repair</option>
				<option value="Exec-managerial">Exec-managerial</option>
				<option value="Farming-fishing">Farming-fishing</option>
				<option value="Handlers-cleaners">Handlers-cleaners</option>
				<option value="Machine-op-inspct">Machine-op-inspct</option>
				<option value="Other-service">Other-service</option>
				<option value="Priv-house-serv">Priv-house-serv</option>
				<option value="Prof-specialty">Prof-specialty</option>
				<option value="Protective-serv">Protective-serv</option>
				<option value="Sales">Sales</option>
				<option value="Tech-support">Tech-support</option>
				<option value="Transport-moving">Transport-moving</option>
			</select>
		</div>

		<div class="col-md-2 mb-1">人種</div>
		<div class="col-md-2 mb-1" >
			<select class="form-control input-lg mb-1" id="race">
				<option value="Amer-Indian-Eskimo">Amer-Indian-Eskimo</option>
				<option value="Asian-Pac-Islander">Asian-Pac-Islander</option>
				<option value="Black">Black</option>
				<option value="Other">Other</option>
				<option value="White">White</option>
			</select>
		</div>
		
		<div class="col-md-2 mb-1">性別</div>
		<div class="col-md-2 mb-1" >
			<select class="form-control input-lg mb-1" id="sex">
				<option value="Male">Male</option>
				<option value="Female">Female</option>
			</select>
		</div>	
		
	</div>
				
	<div class="row">


		<div class="col-md-2 mb-1">キャピタルゲイン</div>
		<div class="col-md-2 mb-1" >
			<input  type="text" class="form-control input-lg" id="capital-gain" value=0 >
		</div>

		<div class="col-md-2 mb-1">キャピタルロス</div>
		<div class="col-md-2 mb-1" >
			<input  type="text" class="form-control input-lg" id="capital-loss" value=0 >
		</div>

		<div class="col-md-2 mb-1">労働時間</div>
		<div class="col-md-2 mb-1" >
			<input  type="text" class="form-control input-lg" id="hours-per-week" value=40 >
		</div>
		
	</div>
			
	<div class="row">

		<div class="col-md-1 mb-1">国</div>
		<div class="col-md-3 mb-1" >
			<select class="form-control input-lg mb-1" id="native-country">
 				<option value="Cambodia">Cambodia</option>
  				<option value="Canada">Canada</option>
  				<option value="China">China</option>
  				<option value="Columbia">Columbia</option>
  				<option value="Cuba">Cuba</option>
 				<option value="Dominican-Republic">Dominican-Republic</option>
 				<option value="Ecuador">Ecuador</option>
  				<option value="El-Salvador">El-Salvador</option>
  				<option value="England">England</option>
  				<option value="France">France</option>
  				<option value="Germany">Germany</option>
 				<option value="Japan">Japan</option>
  				<option value="Laos">Laos</option>
  				<option value="Mexico">Mexico</option>
  				<option value="United-States">United-States</option>
  				<option value="Vietnam">Vietnam</option>
  				<option value="Yugoslavia">Yugoslavia</option>
			</select>
		</div>
		
		<div class="col-md-2 mb-1">アルゴリズム</div>
		<div class="col-md-4 mb-1" >
			<select class="form-control input-lg mb-1" id="alg">
				<option value="1">DecisionTreeClassifier</option>
			</select>
		</div>	
		
		<div class="col-md-2 mb-1" >
			<button class="btn btn-default" id="button" style="width: 100px; padding: 5px;">Load Data</button>
		</div>
		
	</div>

</div>  <!--class="card-body"の終端-->
</div>  <!--class="card"の終端-->

<script type="text/javascript">
"use strict";


var send_data = {};

$("#button").click(function() {

	send_data = JSON.stringify({
	"age":$("#age").val(),
	"workclass":$("#workclass").val(),
	"education-num":$("#education-num").val(),
	"occupation":$("#occupation").val(),
	"race":$("#race").val(),
	"sex":$("#sex").val(),
	"capital-gain":$("#capital-gain").val(),
	"capital-loss":$("#capital-loss").val(),
	"hours-per-week":$("#hours-per-week").val(),
	"native-country":$("#native-country").val()
	});
	
	$.ajax({
		type:'POST',
		url:'/IncomeCal',
		data:send_data,
		contentType:'application/json',
		success:function(data) {
			alert("結果 = " + data );},
		error: function(data) {
			alert('error!!!' + JSON.stringify(data) );
		}
	});
	return false;
});


</script>

{% endblock %}
other
【web.config】
<configuration>
  <appSettings>
    <add key="PYTHONPATH" value="d:\home\site\wwwroot" />
    <add key="WSGI_HANDLER" value="main.application" />
    <add key="WSGI_LOG" value="D:\home\site\wwwroot\LogFiles\wfastcgi.log"/>
  </appSettings>
  <system.webServer>
      <modules runAllManagedModulesForAllRequests="true" />
      <handlers>
        <add name = "Python FastCGI"
             path="*"
             verb="*"
             modules="FastCgiModule"
             scriptProcessor="D:\home\python364x64\python.exe|D:\home\python364x64\wfastcgi.py"
             resourceType="Unspecified"
             requireAccess="Script" />
      </handlers>
      <rewrite>
        <rules>
          <rule name="Static Files" stopProcessing="true">
            <conditions>
              <add input="true" pattern="false" />
            </conditions>
          </rule>
          <rule name="Configure Python" stopProcessing="true">
            <match url="(.*)" ignoreCase="false" />
            <conditions>
              <add input="{REQUEST_URI}" pattern="^/static/.*" ignoreCase="true" negate="true" />
            </conditions>
            <action type="Rewrite" url="handler.fcgi/{R:1}" appendQueryString="true" />
          </rule>
        </rules>
      </rewrite> 
      <httpErrors errorMode="Detailed"></httpErrors>
  </system.webServer>
</configuration>

--------------------------------------
【index.html】
{% extends "base.html" %}
{% block body %}

<div class="container">
  <table class="table table-striped table-condensed">
    <thead>

      <tr>
        <th>NO</th>
        <th>プログラム名</th>
        <th>リンク</th>
        <th>備考</th>
      </tr>
    </thead>
    <tbody>
	

      <tr>      
        <th scope="row">1</th>
        <td>AdultCensusIncomeBinaryClassification dataset識別(画面入力)</td>
        <td>
      			<form action="{{ url_for('Income') }}" style="display: inline" method="post">
      				<button type="submit" class="btn btn-default" style="width: 120px; padding: 5px;">Income</button>
      			</form>
    		</td>
        <td></td>
      </tr>
      
      
      <tr>      
        <th scope="row">2</th>
        <td>scikit-learn識別(画面入力)</td>
        <td>
      			<form action="{{ url_for('Iris') }}" style="display: inline" method="post">
      				<button type="submit" class="btn btn-default" style="width: 120px; padding: 5px;">Iris</button>
      			</form>
    		</td>
        <td></td>
      </tr>
   
	  </tbody>
  </table>
</div>

{% endblock %}

--------------------------------------
【base.html】
<!doctype html>
<!--*****************************************************************
*  共通のhtml
*  
*****************************************************************-->

<head>
	<meta charset="utf-8">
    <title>プロト</title>
	<script src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
	<script src="https://code.jquery.com/ui/1.12.1/jquery-ui.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js"></script>
    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js"></script>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/tabulator/3.5.3/js/tabulator.min.js"></script>

	<script src="{{url_for('static', filename='xlsx.full.min.js')}}"></script>    <!-- cdn設定だとsheetJSが動かない-->
    <script src="{{url_for('static', filename='gsl_common.js')}}"></script>
	
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://oss.sheetjs.com/assets/vendor/samples.css">
    <link rel="stylesheet" href="https://oss.sheetjs.com/assets/css/sheetjs.css">
	<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/tabulator/3.5.3/css/tabulator.min.css">
	<link rel="stylesheet" href="https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.1/themes/smoothness/jquery-ui.css">
	
    <!-- link rel="stylesheet" type=text/css href="{{ url_for('static', filename='style.css') }}" -->

	<style>
		html {overflow-y:scroll ;}
		 
		body {zoom:80% ;}

		*,h1,h2,h3    {font-family:Times New Roman,Meiryo;font-style:italic}

	</style>
	
</head>

<body>
<div class="container">
	<header>
		<!--ナビゲーションメニュー（Navbar）  -->
		<div class="navbar navbar-dark" style="height:40px; background-color:black; ">
			<div class="navbar-brand d-flex align-items-center">
				<strong>Flask(Powerd By Python+Flask+SQLite+Bootstrap4+scikit-learn)</strong>
			</div>
		</div>
	</header>
</div>
	
<div class="container">
    {% block body %}
    {% endblock %}
</div>

</body>
</html>
--------------------------------------
【common.py】
import io
import os
import sys
import platform
import datetime
import time


from flask import *

import json

import sqlite3

import werkzeug
import urllib.request

import pickle

import numpy as np

#画面に表示しようとした時
#from PIL import Image  #Python Image Libraryインストールは pip install Pillow

#**************************************
# 共通処理：SQLiteの戻り値が単純な配列なのでそれを辞書型にする定義
#**************************************
def dict_factory(cursor, row):
    d = {}
    for idx, col in enumerate(cursor.description):
        d[col[0]] = row[idx]
    return d
#**************************************
# ログ出力  if os.path.exists('err.log'):  else:
#**************************************
def log(msg):
    errlog = open('./LogFiles/err.log','a')
    errlog.write( msg + '\n')
    errlog.close()

これでJupyter Notebookで作成したモデルをWebアプリで利用する事が可能になりました。

2
3
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
2
3

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?