3
5

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

Kaggle House Prices ① ~ 特徴量エンジニアリング ~

Last updated at Posted at 2020-10-05

以下の Kaggle のコンペの特徴量エンジニアリングとなります。
House Prices: Advanced Regression Techniques

以下を参考にしています。
kaggleの住宅価格予測問題(House Prices)を解いてみる
kaggle:House Price チュートリアル(住宅価格の予測)

ライブラリの読み込み

import numpy as np
import pandas as pd
from sklearn.externals import joblib

データの読み込み

train = pd.read_csv('train.csv')
test_x = pd.read_csv('test.csv')

外れ値を除外

#物件の広さを合計した変数を作成
train["TotalSF"] = train["1stFlrSF"] + train["2ndFlrSF"] + train["TotalBsmtSF"]
test_x["TotalSF"] = test_x["1stFlrSF"] + test_x["2ndFlrSF"] + test_x["TotalBsmtSF"]
#外れ値を除外する
train = train.drop(train[(train['TotalSF']>7500) & (train['SalePrice']<300000)].index)
train = train.drop(train[(train['YearBuilt']<2000) & (train['SalePrice']>600000)].index)
train = train.drop(train[(train['OverallQual']<5) & (train['SalePrice']>200000)].index)
train = train.drop(train[(train['OverallQual']<10) & (train['SalePrice']>500000)].index)

学習データを目的変数とそれ以外に分ける

train_x = train.drop('SalePrice',axis=1)
train_y = train["SalePrice"]

学習データとテストデータを統合

all_data = pd.concat([train_x,test_x],axis=0,sort=True)

IDのカラムは不必要なので別の変数に格納

train_ID = train_x['Id']
test_ID = test_x['Id']

all_data.drop("Id", axis = 1, inplace = True)

欠損値対応

# 欠損値があるカラムをリスト化
na_col_list = all_data.isnull().sum()[all_data.isnull().sum()>0].index.tolist()

# 隣接した道路の長さ(LotFrontage)の欠損値の補完
all_data['LotFrontage'] = all_data.groupby('Neighborhood')['LotFrontage'].transform(lambda x: x.fillna(x.median()))

# 欠損値が存在するかつfloat型のリストを作成
float_list = all_data[na_col_list].dtypes[all_data[na_col_list].dtypes == "float64"].index.tolist()

# 欠損値が存在するかつobject型のリストを作成
obj_list = all_data[na_col_list].dtypes[all_data[na_col_list].dtypes == "object"].index.tolist()

# float型の場合は欠損値を0で置換
all_data[float_list] = all_data[float_list].fillna(0)

# object型の場合は欠損値を"None"で置換
all_data[obj_list] = all_data[obj_list].fillna("None")

数値変数をカテゴリ変数に変換

all_data['MSSubClass'] = all_data['MSSubClass'].apply(str)
all_data['YrSold'] = all_data['YrSold'].astype(str)
all_data['MoSold'] = all_data['MoSold'].astype(str)

新たな特徴量の追加

# 物件の広さを合計した変数を作成
all_data["TotalSF"] = all_data["1stFlrSF"] + all_data["2ndFlrSF"] + all_data["TotalBsmtSF"]

# 特徴量に1部屋あたりの面積を追加
all_data["FeetPerRoom"] =  all_data["TotalSF"]/all_data["TotRmsAbvGrd"]

# 建築した年とリフォームした年の合計
all_data['YearBuiltAndRemod']=all_data['YearBuilt']+all_data['YearRemodAdd']

# バスルームの合計面積
all_data['Total_Bathrooms'] = (all_data['FullBath'] + (0.5 * all_data['HalfBath']) +
                               all_data['BsmtFullBath'] + (0.5 * all_data['BsmtHalfBath']))

# 縁側の合計面積
all_data['Total_porch_sf'] = (all_data['OpenPorchSF'] + all_data['3SsnPorch'] +
                              all_data['EnclosedPorch'] + all_data['ScreenPorch'] +
                              all_data['WoodDeckSF'])

# プールの有無
all_data['haspool'] = all_data['PoolArea'].apply(lambda x: 1 if x > 0 else 0)

# 2階の有無
all_data['has2ndfloor'] = all_data['2ndFlrSF'].apply(lambda x: 1 if x > 0 else 0)

# ガレージの有無
all_data['hasgarage'] = all_data['GarageArea'].apply(lambda x: 1 if x > 0 else 0)

# 地下室の有無
all_data['hasbsmt'] = all_data['TotalBsmtSF'].apply(lambda x: 1 if x > 0 else 0)

# 暖炉の有無
all_data['hasfireplace'] = all_data['Fireplaces'].apply(lambda x: 1 if x > 0 else 0)

カテゴリ変数に one-hot-encoding を行う

# カテゴリ変数となっているカラムを取り出す
cal_list = all_data.dtypes[all_data.dtypes=="object"].index.tolist()

# カテゴリ変数をget_dummiesによるone-hot-encodingを行う
all_data = pd.get_dummies(all_data,columns=cal_list)

学習データとテストデータに再分割

train_x = all_data.iloc[:train_x.shape[0],:].reset_index(drop=True)
test_x = all_data.iloc[train_x.shape[0]:,:].reset_index(drop=True)

特徴量を保存

joblib.dump(train_x, 'train_x.pkl')
joblib.dump(test_x, 'test_x.pkl')
joblib.dump(train_y, 'train_y.pkl')
3
5
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
3
5

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?