探索的データ分析
Explanatory Data Analysisの重要性は
EDA ツール
CSVデータの例でEDAに関するデータの可視化のツールをまとめます。
リクワイアメント
- Download https://www.kaggle.com/c/titanic/data?select=train.csv
- Install
pip install pandas
Dataprep
Install
pip install dataprep
Usage
import pandas as pd
from dataprep.eda import create_report
train = pd.read_csv('train.csv')
create_report(train).show_browser()
レポートのSceenshot
Pandas Profiling
Install
pip install pandas-profiling
Usage
import pandas as pd
from pandas_profiling import ProfileReport
train = pd.read_csv('train.csv')
profile = ProfileReport(train, title="Report")
profile
# Save as a HTML file
# profile.to_file("pandas_profiling_train.html")
レポートのSceenshot
Sweetviz
Install
pip install sweetviz
Usage
import sweetviz as sv
train = pd.read_csv('train.csv')
analyze_report = sv.analyze(train)
analyze_report.show_html('report.html', open_browser=True)
レポートのSceenshot
AutoViz
Install
pip install autoviz
Usage
from autoviz.AutoViz_Class import AutoViz_Class
AV = AutoViz_Class()
df_av = AV.AutoViz('train.csv', chart_format='bokeh')
# Local web server
#df_av = AV.AutoViz('train.csv', chart_format='server')
# Save charts as HTML files at AutoViz_Plots/
#df_av = AV.AutoViz('train.csv', chart_format='html')
レポートのSceenshot
PipeRider
CLIでEDAとAssertionするツールです
Install
pip install 'piperider[csv]'
CLI Usage
piperider init
Initialize piperider to path /Users/gabriel/Workspace/playground/titanic/.piperider
[?] What is your data source name? (alphanumeric and underscore are allowed): titanic
[?] Which data source would you like to connect to?: csv
> csv
Please enter the following fields for csv
[?] Path of csv file: train.csv
piperider run
[?] Do you want to auto generate recommended assertions for this datasource [Yes/no]? Yes
Generating reports from: ~/titanic/.piperider/outputs/latest/run.json
Report generated in ~/titanic/.piperider/outputs/latest/index.html
レポートのSceenshot
Whylogs （追記）
Install
# With Profile Visualizer
pip install 'whylogs[viz]'
Usage
import whylogs as why
import pandas as pd
#dataframe
train = pd.read_csv("train.csv")
result = why.log(pandas=train)
train_view = result.view()
from whylogs.viz import NotebookProfileVisualizer
visualization = NotebookProfileVisualizer()
visualization.set_profiles(train_view)
visualization.profile_summary()
レポートのSceenshot
Data Profiler（追記）
AIでEDAすることができるここのProfilerですが、HTMLのレポートがありません。
Install
# Report only
pip install DataProfiler[report]
# With Tensorflow
pip install DataProfiler[ml]
# Full package
pip install DataProfiler[full]
Usage
import json
from dataprofiler import Data, Profiler
data = Data("train.csv") # Auto-Detect & Load: CSV, AVRO, Parquet, JSON, Text, URL
print(data.data.head(5)) # Access data directly via a compatible Pandas DataFrame
profile = Profiler(data) # Calculate Statistics, Entity Recognition, etc
readable_report = profile.report(report_options={"output_format": "compact"})
print(json.dumps(readable_report, indent=4))
レポートのJSON
{
"global_stats": {
"samples_used": 891,
"column_count": 12,
"row_count": 891,
"row_has_null_ratio": 0.7946,
"row_is_null_ratio": 0.0,
"unique_row_ratio": 1.0,
"duplicate_row_count": 0,
"file_type": "csv",
"encoding": "utf-8",
"correlation_matrix": null,
"chi2_matrix": "[[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan], ... , [nan, 0., 0., nan, 0., 0., 0., 0., nan, nan, nan, 1.]]",
"profile_schema": {
"PassengerId": [
0
],
"Survived": [
1
],
"Pclass": [
2
],
"Name": [
3
],
"Sex": [
4
],
"Age": [
5
],
"SibSp": [
6
],
"Parch": [
7
],
"Ticket": [
8
],
"Fare": [
9
],
"Cabin": [
10
],
"Embarked": [
11
]
},
"times": {
"row_stats": 0.003
}
},
"data_stats": [
{
"column_name": "PassengerId",
"data_type": "int",
"categorical": false,
"order": "ascending",
"samples": "['600', '74', '831', '76', '811']",
"statistics": {
"min": 1.0,
"max": 891.0,
"mode": "[1.445, 2.335, 3.225, 4.115, 5.005]",
"median": 446.445,
"sum": 397386.0,
"mean": 446.0,
"variance": 66231.0,
"stddev": 257.3538,
"skewness": 0.0,
"kurtosis": -1.2,
"quantiles": {
"0": 223.2775,
"1": 446.445,
"2": 668.7225
},
"median_abs_deviation": 222.7225,
"num_zeros": 0,
"num_negatives": 0,
"unique_count": 891,
"unique_ratio": 1.0,
"sample_size": 891,
"null_count": 0,
"null_types": "[]",
"data_type_representation": {
"datetime": 0.0,
"int": 1.0,
"float": 1.0,
"string": 1.0
}
}
},
{
"column_name": "Survived",
"data_type": "int",
"categorical": true,
"order": "random",
"samples": "['1', '0', '0', '1', '1']",
"statistics": {
"min": 0.0,
"max": 1.0,
"mode": "[0.0005]",
"median": 0.0008,
"sum": 342.0,
"mean": 0.3838,
"variance": 0.2368,
"stddev": 0.4866,
"skewness": 0.4785,
"kurtosis": -1.775,
"quantiles": {
"0": 0.0004,
"1": 0.0008,
"2": 0.9993
},
"median_abs_deviation": 0,
"num_zeros": 549,
"num_negatives": 0,
"unique_count": 2,
"unique_ratio": 0.0022,
"categories": "['0', '1']",
"gini_impurity": 0.473,
"unalikeability": 0.4735,
"categorical_count": {
"0": 549,
"1": 342
},
"sample_size": 891,
"null_count": 0,
"null_types": "[]",
"data_type_representation": {
"datetime": 0.0,
"int": 1.0,
"float": 1.0,
"string": 1.0
}
}
},
{
"column_name": "Pclass",
"data_type": "int",
"categorical": true,
"order": "random",
"samples": "['2', '2', '1', '1', '3']",
"statistics": {
"min": 1.0,
"max": 3.0,
"mode": "[2.999]",
"median": 2.9982,
"sum": 2057.0,
"mean": 2.3086,
"variance": 0.699,
"stddev": 0.8361,
"skewness": -0.6305,
"kurtosis": -1.28,
"quantiles": {
"0": 2.0001,
"1": 2.9982,
"2": 2.9991
},
"median_abs_deviation": 0.0016,
"num_zeros": 0,
"num_negatives": 0,
"unique_count": 3,
"unique_ratio": 0.0034,
"categories": "['3', '1', '2']",
"gini_impurity": 0.5949,
"unalikeability": 0.5956,
"categorical_count": {
"3": 491,
"1": 216,
"2": 184
},
"sample_size": 891,
"null_count": 0,
"null_types": "[]",
"data_type_representation": {
"datetime": 0.0,
"int": 1.0,
"float": 1.0,
"string": 1.0
}
}
},
{
"column_name": "Name",
"data_type": "string",
"categorical": false,
"order": "random",
"samples": "['Smiljanic, Mr. Mile', 'Isham, Miss. Ann Elizabeth',\n 'Petranec, Miss. Matilda', 'Ling, Mr. Lee',\n 'Kirkland, Rev. Charles Leonard']",
"statistics": {
"min": 12.0,
"max": 82.0,
"mode": "[19.035]",
"median": 25.0041,
"sum": 24026.0,
"mean": 26.9652,
"variance": 86.1482,
"stddev": 9.2816,
"skewness": 1.3926,
"kurtosis": 2.5594,
"quantiles": {
"0": 20.0137,
"1": 25.0041,
"2": 30.0586
},
"median_abs_deviation": 5.0216,
"vocab": "['O', ' ', 'k', 'a', 'P', ... , 'K', 'j', 'd', 'Q', 'x']",
"unique_count": 891,
"unique_ratio": 1.0,
"sample_size": 891,
"null_count": 0,
"null_types": "[]",
"data_type_representation": {
"datetime": 0.0,
"int": 0.0,
"float": 0.0,
"string": 1.0
}
}
},
{
"column_name": "Sex",
"data_type": "string",
"categorical": true,
"order": "random",
"samples": "['male', 'male', 'female', 'male', 'male']",
"statistics": {
"min": 4.0,
"max": 6.0,
"mode": "[4.001]",
"median": 4.0015,
"sum": 4192.0,
"mean": 4.7048,
"variance": 0.9139,
"stddev": 0.956,
"skewness": 0.6189,
"kurtosis": -1.6206,
"quantiles": {
"0": 4.0008,
"1": 4.0015,
"2": 5.9986
},
"median_abs_deviation": 0,
"vocab": "['e', 'm', 'a', 'f', 'l']",
"unique_count": 2,
"unique_ratio": 0.0022,
"categories": "['male', 'female']",
"gini_impurity": 0.4564,
"unalikeability": 0.4569,
"categorical_count": {
"male": 577,
"female": 314
},
"sample_size": 891,
"null_count": 0,
"null_types": "[]",
"data_type_representation": {
"datetime": 0.0,
"int": 0.0,
"float": 0.0,
"string": 1.0
}
}
},
{
"column_name": "Age",
"data_type": "float",
"categorical": true,
"order": "random",
"samples": "['28', '34', '27', '26', '63']",
"statistics": {
"min": 0.42,
"max": 80.0,
"mode": "[24.01547]",
"median": 28.0183,
"sum": 21205.17,
"mean": 29.6991,
"variance": 211.0191,
"stddev": 14.5265,
"skewness": 0.3891,
"kurtosis": 0.1783,
"quantiles": {
"0": 20.0736,
"1": 28.0183,
"2": 38.0505
},
"median_abs_deviation": 8.9421,
"num_zeros": 0,
"num_negatives": 0,
"precision": {
"min": 1,
"max": 3,
"mean": 1.85,
"var": 0.18,
"std": 0.424,
"sample_size": 714,
"margin_of_error": 0.0523,
"confidence_level": 0.999
},
"unique_count": 88,
"unique_ratio": 0.1232,
"categories": "['24', '22', '18', '19', ... , '55.5', '0.92', '23.5', '74']",
"gini_impurity": 0.978,
"unalikeability": 0.9794,
"categorical_count": {
"24": 30,
"22": 27,
"18": 26,
"19": 25,
"28": 25,
"30": 25,
"21": 24,
"25": 23,
"36": 22,
"29": 20,
"32": 18,
"35": 18,
"27": 18,
"26": 18,
"16": 17,
"31": 17,
"20": 15,
"34": 15,
"33": 15,
"23": 15,
"39": 14,
"40": 13,
"17": 13,
"42": 13,
"45": 12,
"38": 11,
"50": 10,
"2": 10,
"4": 10,
"44": 9,
"48": 9,
"47": 9,
"54": 8,
"9": 8,
"1": 7,
"51": 7,
"14": 6,
"52": 6,
"37": 6,
"49": 6,
"41": 6,
"3": 6,
"58": 5,
"15": 5,
"43": 5,
"62": 4,
"56": 4,
"5": 4,
"11": 4,
"60": 4,
"8": 4,
"6": 3,
"46": 3,
"61": 3,
"65": 3,
"7": 3,
"10": 2,
"64": 2,
"13": 2,
"63": 2,
"30.5": 2,
"57": 2,
"70": 2,
"0.75": 2,
"71": 2,
"59": 2,
"0.83": 2,
"40.5": 2,
"55": 2,
"32.5": 2,
"28.5": 2,
"45.5": 2,
"34.5": 1,
"0.42": 1,
"0.67": 1,
"66": 1,
"24.5": 1,
"80": 1,
"20.5": 1,
"53": 1,
"14.5": 1,
"70.5": 1,
"12": 1,
"36.5": 1,
"55.5": 1,
"0.92": 1,
"23.5": 1,
"74": 1
},
"sample_size": 891,
"null_count": 177,
"null_types": "['']",
"data_type_representation": {
"datetime": 0.0,
"int": 0.965,
"float": 1.0,
"string": 1.0
}
}
},
{
"column_name": "SibSp",
"data_type": "int",
"categorical": true,
"order": "random",
"samples": "['1', '0', '0', '5', '0']",
"statistics": {
"min": 0.0,
"max": 8.0,
"mode": "[0.004]",
"median": 0.0059,
"sum": 466.0,
"mean": 0.523,
"variance": 1.216,
"stddev": 1.1027,
"skewness": 3.6954,
"kurtosis": 17.8804,
"quantiles": {
"0": 0.0029,
"1": 0.0059,
"2": 1.0023
},
"median_abs_deviation": 0,
"num_zeros": 608,
"num_negatives": 0,
"unique_count": 7,
"unique_ratio": 0.0079,
"categories": "['0', '1', '2', '4', '3', '8', '5']",
"gini_impurity": 0.4775,
"unalikeability": 0.4781,
"categorical_count": {
"0": 608,
"1": 209,
"2": 28,
"4": 18,
"3": 16,
"8": 7,
"5": 5
},
"sample_size": 891,
"null_count": 0,
"null_types": "[]",
"data_type_representation": {
"datetime": 0.0,
"int": 1.0,
"float": 1.0,
"string": 1.0
}
}
},
{
"column_name": "Parch",
"data_type": "int",
"categorical": true,
"order": "random",
"samples": "['1', '0', '0', '0', '0']",
"statistics": {
"min": 0.0,
"max": 6.0,
"mode": "[0.003]",
"median": 0.0039,
"sum": 340.0,
"mean": 0.3816,
"variance": 0.6497,
"stddev": 0.8061,
"skewness": 2.7491,
"kurtosis": 9.7781,
"quantiles": {
"0": 0.002,
"1": 0.0039,
"2": 0.0059
},
"median_abs_deviation": 0,
"num_zeros": 678,
"num_negatives": 0,
"unique_count": 7,
"unique_ratio": 0.0079,
"categories": "['0', '1', '2', '5', '3', '4', '6']",
"gini_impurity": 0.3953,
"unalikeability": 0.3957,
"categorical_count": {
"0": 678,
"1": 118,
"2": 80,
"5": 5,
"3": 5,
"4": 4,
"6": 1
},
"sample_size": 891,
"null_count": 0,
"null_types": "[]",
"data_type_representation": {
"datetime": 0.0,
"int": 1.0,
"float": 1.0,
"string": 1.0
}
}
},
{
"column_name": "Ticket",
"data_type": "string",
"categorical": false,
"order": "random",
"samples": "['113781', '315097', '371110', 'C.A. 17248', '36947']",
"statistics": {
"min": 3.0,
"max": 18.0,
"mode": "[6.0075]",
"median": 6.0076,
"sum": 6015.0,
"mean": 6.7508,
"variance": 7.5379,
"stddev": 2.7455,
"skewness": 2.211,
"kurtosis": 5.1754,
"quantiles": {
"0": 5.0087,
"1": 6.0076,
"2": 6.9985
},
"median_abs_deviation": 0.9972,
"vocab": "['H', '5', '7', 'Q', '3', ... , '0', 'P', 'e', 'L', 'N']",
"unique_count": 681,
"unique_ratio": 0.7643,
"sample_size": 891,
"null_count": 0,
"null_types": "[]",
"data_type_representation": {
"datetime": 0.4837,
"int": 0.7419,
"float": 0.7419,
"string": 1.0
}
}
},
{
"column_name": "Fare",
"data_type": "float",
"categorical": false,
"order": "random",
"samples": "['7.8958', '9.5', '0', '34.375', '227.525']",
"statistics": {
"min": 0.0,
"max": 512.3292,
"mode": "[7.9411026]",
"median": 14.5475,
"sum": 28693.9493,
"mean": 32.2042,
"variance": 2469.4368,
"stddev": 49.6934,
"skewness": 4.7873,
"kurtosis": 33.3981,
"quantiles": {
"0": 8.0222,
"1": 14.5475,
"2": 31.124
},
"median_abs_deviation": 6.945,
"num_zeros": 15,
"num_negatives": 0,
"precision": {
"min": 0,
"max": 7,
"mean": 3.899,
"var": 2.3898,
"std": 1.5459,
"sample_size": 891,
"margin_of_error": 0.1704,
"confidence_level": 0.999
},
"unique_count": 248,
"unique_ratio": 0.2783,
"sample_size": 891,
"null_count": 0,
"null_types": "[]",
"data_type_representation": {
"datetime": 0.0,
"int": 0.1807,
"float": 1.0,
"string": 1.0
}
}
},
{
"column_name": "Cabin",
"data_type": "string",
"categorical": false,
"order": "random",
"samples": "['B42', 'E25', 'D', 'E33', 'A23']",
"statistics": {
"min": 1.0,
"max": 15.0,
"mode": "[2.995]",
"median": 2.9961,
"sum": 732.0,
"mean": 3.5882,
"variance": 4.3025,
"stddev": 2.0743,
"skewness": 3.1847,
"kurtosis": 11.7603,
"quantiles": {
"0": 2.9905,
"1": 2.9961,
"2": 3.0017
},
"median_abs_deviation": 0.0056,
"vocab": "['5', '7', '3', 'T', 'E', ... , '1', 'D', '4', ' ', '0']",
"unique_count": 147,
"unique_ratio": 0.7206,
"sample_size": 891,
"null_count": 687,
"null_types": "['']",
"data_type_representation": {
"datetime": 0.0,
"int": 0.0,
"float": 0.0,
"string": 1.0
}
}
},
{
"column_name": "Embarked",
"data_type": "string",
"categorical": true,
"order": "random",
"samples": "['S', 'S', 'S', 'S', 'S']",
"statistics": {
"min": 1.0,
"max": 1.0,
"mode": "[1.]",
"median": 1.0,
"sum": 889.0,
"mean": 1.0,
"variance": 0.0,
"stddev": 0.0,
"skewness": 0.0,
"kurtosis": -3.0102,
"quantiles": {
"0": 1.0,
"1": 1.0,
"2": 1.0
},
"median_abs_deviation": 0,
"vocab": "['S', 'Q', 'C']",
"unique_count": 3,
"unique_ratio": 0.0034,
"categories": "['S', 'C', 'Q']",
"gini_impurity": 0.432,
"unalikeability": 0.4325,
"categorical_count": {
"S": 644,
"C": 168,
"Q": 77
},
"sample_size": 891,
"null_count": 2,
"null_types": "['']",
"data_type_representation": {
"datetime": 0.0,
"int": 0.0,
"float": 0.0,
"string": 1.0
}
}
}
]
}
Lux（追記）
Install
pip install lux-api
Usage
import lux
import pandas as pd
df = pd.read_csv("train.csv")
df
# df.save_as_html("lux.html")
レポートのSceenshot
Polymersearch（追記）
Webでspreadsheetやcsvを読み込んでEDAします
本文記載時点は無料で14-days-trialがあります
レポートのSceenshot
