Canopy Expressをインストール
https://store.enthought.com/downloads/
canopy-1.5.2-osx-64.dmg
300MBぐらい

ドラッグ・アンド・ドロップでインストールして、実行許可を与えたら初期設定が始まる。

デフォルトのPythonを変えてよい？と聞かれYes。

iTermでpythonと打ってもどうも変わっていない。

% which python
/usr/bin/python

.bash_profileに追加されている

# Added by Canopy installer on 2015-02-15
# VIRTUAL_ENV_DISABLE_PROMPT can be set to '' to make bashprompt show that Canopy is active, otherwise 1
VIRTUAL_ENV_DISABLE_PROMPT=1 source /Users/takeru/Library/Enthought/Canopy_64bit/User/bin/activate

zshだからだ。

% VIRTUAL_ENV_DISABLE_PROMPT=1 source /Users/takeru/Library/Enthought/Canopy_64bit/User/bin/activate
% which python
/Users/takeru/Library/Enthought/Canopy_64bit/User/bin/python

変わった。
単にそれを実行すればPATHを変えてくれるようだ。

なんか出た！

http://www.usa.gov/About/developer-resources/1usagov.shtml#How_to_Access_The_Data
http://1usagov.measuredvoice.com/2013/
なんか更新止まってるみたいだけど古いのは取れる。

20:29:59 tkrimac2:~/proj/pydata% gunzip usagov_bitly_data2013-05-17-1368832207.gz
20:30:03 tkrimac2:~/proj/pydata% ls
usagov_bitly_data2013-05-17-1368832207

20:30:23 tkrimac2:~/proj/pydata% ipython --pylab
Python 2.7.6 | 64-bit | (default, Sep 15 2014, 17:43:19)
Type "copyright", "credits" or "license" for more information.

IPython 2.3.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
Using matplotlib backend: MacOSX

In [1]: path="usagov_bitly_data2013-05-17-1368832207"

In [2]: import json

In [3]: records = [json.loads(line) for line in open(path)]

In [4]: records[0]
Out[4]:
{u'a': u'Mozilla/5.0 (Linux; U; Android 4.1.2; en-us; HTC_PN071 Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30',
 u'al': u'en-US',
 u'c': u'US',
 u'cy': u'Anaheim',
 u'g': u'15r91',
 u'gr': u'CA',
 u'h': u'10OBm3W',
 u'hc': 1365701422,
 u'hh': u'j.mp',
 u'l': u'pontifier',
 u'll': [33.816101, -117.979401],
 u'nk': 0,
 u'r': u'direct',
 u't': 1368832205,
 u'tz': u'America/Los_Angeles',
 u'u': u'http://www.nsa.gov/'}

In [5]: records[0]['tz']
Out[5]: u'America/Los_Angeles'

In [6]: print records[0]['tz']
America/Los_Angeles

In [11]: time_zones = [rec['tz'] for rec in records if 'tz' in rec]

In [12]: time_zones[:10]
Out[12]:
[u'America/Los_Angeles',
 u'',
 u'America/Phoenix',
 u'America/Chicago',
 u'',
 u'America/Indianapolis',
 u'America/Chicago',
 u'',
 u'Australia/NSW',
 u'']

tmp.pyを作って

# tmp.py
print "a"

In [18]: import tmp
a

(tmp.pyの"a"を"aaa"に書き換え)

In [19]: import tmp
(読み込まれない)

In [23]: reload(tmp)
aaa

In [27]: reload(tmp)
Out[27]: <module 'tmp' from 'tmp.py'>

In [28]: tmp.get_counts(time_zones)
Out[28]:
{u'': 636,
 u'Africa/Cairo': 3,
 u'Africa/Casablanca': 1,
 u'Africa/Ceuta': 6,
 u'Africa/Gaborone': 1,
 u'Africa/Johannesburg': 2,
 u'America/Anchorage': 8,
 u'America/Argentina/Buenos_Aires': 11,
...

In [29]: reload(tmp)
Out[29]: <module 'tmp' from 'tmp.py'>

In [30]: tmp.get_counts2(time_zones)
Out[30]: defaultdict(<type 'int'>, {u'': 636, u'Europe/Lisbon': 8, u'America/Bogota': 16, u'America/Edmonton': 9, u'Australia/Tasmania': 1, u'Europe/Tallinn': 1, u'Asia/Calcutta': 6, u'Australia/South': 4, u'Europe/Skopje': 1, u'Europe/Copenhagen': 4, u'America/St_Lucia': 1, u'Europe/Amsterdam': 15, u'Europe/Zaporozhye': 1, u'America/Phoenix': 40, u'Europe/Moscow': 35, u'America/El_Salvador': 2, u'Europe/Madrid': 21,

In [31]: counts = tmp.get_counts(time_zones)

In [32]: counts["Asia/Tokyo"]
Out[32]: 102

In [33]: counts2 = tmp.get_counts2(time_zones)

In [34]: counts["Asia/Tokyo"]
Out[34]: 102

In [37]: tmp.top_counts(counts)
Out[37]:
[(40, u'America/Phoenix'),
 (50, u'America/Indianapolis'),
 (85, u'Europe/London'),
 (89, u'America/Denver'),
 (102, u'Asia/Tokyo'),
 (184, u'America/Puerto_Rico'),
 (421, u'America/Los_Angeles'),
 (636, u''),
 (686, u'America/Chicago'),
 (903, u'America/New_York')]

# tmp.py

def get_counts(sequence):
	counts = {}
	for x in sequence:
		if x in counts:
			counts[x] += 1
		else:
			counts[x] = 1
	return counts



from collections import defaultdict

def get_counts2(sequence):
	counts = defaultdict(int)
	for x in sequence:
		counts[x] += 1
	return counts


def top_counts(count_dict, n=10):
	value_key_pairs = [(count,tz) for tz,count in count_dict.items()]
	value_key_pairs.sort()
	return value_key_pairs[-n:]


from collections import Counter

In [41]: counts3 = tmp.Counter(time_zones)

In [42]: counts3
Out[42]: Counter({u'America/New_York': 903, u'America/Chicago': 686, u'': 636, u'America/Los_Angeles': 421, u'America/Puerto_Rico': 184, u'Asia/Tokyo': 102, u'America/Denver': 89, u'Europe/London': 85, u'America/Indianapolis': 50, u'America/Phoenix': 40, u'Europe/Moscow': 35, u'America/Rainy_River': 33, u'Australia/NSW': 32, u'America/Sao_Paulo': 29, u'Europe/Paris': 27, u'Europe/Berlin': 24, u'America/Vancouver': 23,

In [43]: counts3.most_common(10)
Out[43]:
[(u'America/New_York', 903),
 (u'America/Chicago', 686),
 (u'', 636),
 (u'America/Los_Angeles', 421),
 (u'America/Puerto_Rico', 184),
 (u'Asia/Tokyo', 102),
 (u'America/Denver', 89),
 (u'Europe/London', 85),
 (u'America/Indianapolis', 50),
 (u'America/Phoenix', 40)]

In [44]: from pandas import DataFrame,Series

In [45]: import pandas as pd; import numpy as np

In [46]: frame = DataFrame(records)

In [47]: frame
Out[47]:
      _heartbeat_                                                  a  \
0             NaN  Mozilla/5.0 (Linux; U; Android 4.1.2; en-us; H...
1             NaN  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT ...
2             NaN  Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20...
3             NaN  Mozilla/5.0 (Linux; U; Android 4.1.2; en-us; S...
4             NaN  Opera/9.80 (Android; Opera Mini/7.5.33286/29.3...
5             NaN  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT...
6             NaN  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) G...
7             NaN  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_5 li...
8             NaN  Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_3 like ...
9             NaN  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...
10            NaN  Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_3 like ...
11            NaN  Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_3 like ...
12            NaN  Mozilla/5.0 (iPad; CPU OS 6_1_2 like Mac OS X)...
13            NaN  Mozilla/5.0 (Windows NT 5.1; rv:20.0) Gecko/20...
14            NaN  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT ...
15            NaN  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; r...
16            NaN  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...
17            NaN  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...
18            NaN  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3)...
19            NaN  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) G...
20            NaN  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT ...
21            NaN  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...
22            NaN  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...
23            NaN  Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; L...
24            NaN  Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.3...
25            NaN  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT ...
26            NaN  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT ...
27            NaN  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT ...
28            NaN  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT ...
29            NaN  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT ...
...           ...                                                ...
3929          NaN  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...
3930          NaN  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; r...
3931          NaN  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3)...
3932          NaN  Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.3...
3933          NaN  Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_3 like ...
3934          NaN  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) G...
3935          NaN  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...
3936          NaN  Mozilla/5.0 (compatible; Genieo/1.0 http://www...
3937          NaN  Mozilla/5.0 (Linux; U; Android 4.0.3; en-gb; H...
3938          NaN  Mozilla/5.0 (Linux; U; Android 4.1.2; es-es; G...
3939          NaN                                 ShortLinkTranslate
3940          NaN  Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; L...
3941          NaN  Mozilla/5.0 (iPad; CPU OS 6_1_3 like Mac OS X)...
3942          NaN  Mozilla/5.0 (iPad; CPU OS 6_1_3 like Mac OS X)...
3943          NaN  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT ...
3944          NaN  Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; D...
3945          NaN  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) G...
3946          NaN  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.3...
3947          NaN  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3)...
3948          NaN  Mozilla/5.0 (Linux; Android 4.0.4; SO-03D Buil...
3949          NaN  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) G...
3950          NaN  Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_4 like ...
3951          NaN  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...
3952          NaN  Mozilla/5.0 (iPad; CPU OS 6_1_3 like Mac OS X)...
3953   1368835801                                                NaN
3954          NaN  Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; M...
3955          NaN  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3)...
3956          NaN                                 ShortLinkTranslate
3957          NaN  Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_4 like ...
3958          NaN  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...

                                       al     c                   cy        g  \
0                                   en-US    US              Anaheim    15r91
1                                   en-us  None                  NaN   ifIpBW
2                          en-US,en;q=0.5    US        Fort Huachuca  10DaxOu
3                                   en-US    US              Houston   TysVFU
4                                      en  None                  NaN  10IGW7m
5                                   en-US    US            Mishawaka  13GrCeP
6                          en-US,en;q=0.5    US              Hammond   YmtpnZ
7                                   en-us  None                  NaN  13oM0hV
8                                   en-us    AU               Sydney    15r91
9                          en-US,en;q=0.8  None                  NaN  109LtDc
10                                  en-us    US           Middletown  109ar5F
11                                  en-us    US           Germantown  107xZnW
12                                  en-us    US             Richmond  19AcekS
13                         en-US,en;q=0.5    US             Portland  16mY628
14                                  en-us    US               Aurora   YRyW8K
15                         en-US,en;q=0.5    US              Houston  18NUp44
16                         en-US,en;q=0.8    US              Muskego   YmtpnZ
17                                  en-us    US               Arvada   ZPictr
18                                  en-us    US                 Bend  11C6yJk
19                         en-US,en;q=0.5    US               Laurel  15RP5hF
20                                  en-us    US              Seattle  12yP2Cx
21                         en-US,en;q=0.8  None                  NaN  109ar5F
22                         en-US,en;q=0.8    US               Durand   YmtpnZ
23                         en-us,en;q=0.9  None                  NaN  107xZnW
24                                    NaN  None                  NaN   YmtpnZ
25                         en-us,en;q=0.5  None                  NaN   ifIpBW
26                         en-us,en;q=0.5  None                  NaN   ifIpBW
27                         en-us,en;q=0.5  None                  NaN   ifIpBW
28                         en-us,en;q=0.5  None                  NaN   ifIpBW
29                         en-us,en;q=0.5  None                  NaN   ifIpBW
...                                   ...   ...                  ...      ...
3929              ja,en-US;q=0.8,en;q=0.6  None                  NaN  10Kc32m
3930  fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3    CH             Chambesy  14bmsHn
3931                                en-us    US            Rockville  14bmsHn
3932                       en-US,en;q=0.8  None                  NaN    15r91
3933                                en-us    US            Ann Arbor  1084Psg
3934                       en-US,en;q=0.5    US          Thomasville    15r91
3935                       en-US,en;q=0.8    US  Coronado Ntl Forest  186NWQK
3936                                  NaN    US      Manhattan Beach   YYv1XQ
3937                         en-GB, en-US  None                  NaN  12AyUk2
3938                         es-ES, en-US  None                  NaN  12AyUk2
3939                                  NaN    JP              Kashiwa   YPnFn4
3940                                en-US  None                  NaN    15r91
3941                                en-us    US           Marshfield   YmtpnZ
3942                       en-US,en;q=0.8    US               Vaughn  16uqtLe
3943                                en-US    US              Orlando  10WMBv9
3944                                en-US    US             Lakewood  14bmsHn
3945                       en-US,en;q=0.5    US                Boone  10X5IW8
3946                       en-US,en;q=0.8  None                  NaN   YmtpnZ
3947                       en-US,en;q=0.8    US                Logan  107xZnW
3948              ja,en-US;q=0.8,en;q=0.6    JP                Tokyo  15TFyGK
3949                       en-gb,en;q=0.5    US        Castro Valley  11C6yJk
3950                                en-us    US         Fayetteville  10ydrrV
3951                                en-us    US                Salem  107xZnW
3952                                en-us    US              Grayson  107xZnW
3953                                  NaN   NaN                  NaN      NaN
3954                                en-US    US               Mobile  10WWSaR
3955                                en-us    US           Brookfield   YmtpnZ
3956                                  NaN    JP              Tsukuba   YPnFn4
3957                                en-us  None                  NaN  17B6VoC
3958                                en-us    PR             Guaynabo  10WMBv9

       gr        h          hc           hh   kw             l  \
0      CA  10OBm3W  1365701422         j.mp  NaN     pontifier
1     NaN   ifIpBW  1302189369    1.usa.gov  NaN         bitly
2      AZ  10DaxOt  1368814585    1.usa.gov  NaN     jaxstrong
3      TX   TChsoQ  1354719206    1.usa.gov  NaN  o_5004fs3lvd
4     NaN  10IGW7l  1368738258    1.usa.gov  NaN    peacecorps
5      IN  13GrCeP  1368130510    1.usa.gov  NaN         bitly
6      WI   YmtpnZ  1363711958    1.usa.gov  NaN         bitly
7     NaN  15PUeH0  1368714329  go.nasa.gov  NaN   nasatwitter
8      02  10OBm3W  1365701422         j.mp  NaN     pontifier
9     NaN  109LtDb  1368821840  go.nasa.gov  NaN   nasatwitter
10     OH  109ar5E  1368803813    1.usa.gov  NaN    usairforce
11     MD  107xZnW  1368815450    1.usa.gov  NaN         bitly
12     KY  19AcekR  1368738410    1.usa.gov  NaN    peacecorps
13     OR  16mY627  1368743779    1.usa.gov  NaN       pbierce
14     IL   YRyW8K  1368475960    1.usa.gov  NaN         bitly
15     TX  18NUoNR  1368727073    1.usa.gov  NaN  o_1fs5ea3lim
16     WI   YmtpnZ  1363711958    1.usa.gov  NaN         bitly
17     CO   ZPictq  1366901428    1.usa.gov  NaN   o_d63rn9enb
18     OR  19oVtZN  1368558078    1.usa.gov  NaN     raylahood
19     MD  16Ewvc4  1368818301    1.usa.gov  NaN       rebroth
20     WA  12yP2Cw  1368741846    1.usa.gov  NaN  o_6vo5h05abv
21    NaN  109ar5E  1368803813    1.usa.gov  NaN    usairforce
22     WI   YmtpnZ  1363711958    1.usa.gov  NaN         bitly
23    NaN  107xZnW  1368815450    1.usa.gov  NaN         bitly
24    NaN   YmtpnZ  1363711958    1.usa.gov  NaN         bitly
25    NaN   ifIpBW  1302189369    1.usa.gov  NaN         bitly
26    NaN   ifIpBW  1302189369    1.usa.gov  NaN         bitly
27    NaN   ifIpBW  1302189369    1.usa.gov  NaN         bitly
28    NaN   ifIpBW  1302189369    1.usa.gov  NaN         bitly
29    NaN   ifIpBW  1302189369    1.usa.gov  NaN         bitly
...   ...      ...         ...          ...  ...           ...
3929  NaN  10Kc32l  1368809020  go.nasa.gov  NaN   nasatwitter
3930   07  14bmsHn  1368223558    1.usa.gov  NaN         bitly
3931   MD  14bmsHn  1368223558    1.usa.gov  NaN         bitly
3932  NaN  10OBm3W  1365701422         j.mp  NaN     pontifier
3933   MI  1084Psg  1368755511         j.mp  NaN         bitly
3934   NC  10OBm3W  1365701422         j.mp  NaN     pontifier
3935   AZ  186NWQK  1368828881    1.usa.gov  NaN         bitly
3936   CA   YYv1XQ  1368710703    1.usa.gov  NaN         bitly
3937  NaN  12AyUk1  1368808362  go.nasa.gov  NaN   nasatwitter
3938  NaN  12AyUk1  1368808362  go.nasa.gov  NaN   nasatwitter
3939   04   YPnFn3  1368833354    1.usa.gov  NaN        hayano
3940  NaN  10OBm3W  1365701422         j.mp  NaN     pontifier
3941   WI   YmtpnZ  1363711958    1.usa.gov  NaN         bitly
3942   WA  16uqtLd  1368455001    1.usa.gov  NaN  o_33avl0ri1b
3943   FL  10WMBv9  1368826755    1.usa.gov  NaN         bitly
3944   OH  14bmsHn  1368223558    1.usa.gov  NaN         bitly
3945   NC  10X5IW7  1368834590    1.usa.gov  NaN          inws
3946  NaN   YmtpnZ  1363711958    1.usa.gov  NaN         bitly
3947   UT  107xZnW  1368815450    1.usa.gov  NaN         bitly
3948   40  15TFyGJ  1368810402  go.nasa.gov  NaN   nasatwitter
3949   CA  19oVtZN  1368558078    1.usa.gov  NaN     raylahood
3950   GA  10ydrrU  1368806410    1.usa.gov  NaN   fsanewmedia
3951   VA  107xZnW  1368815450    1.usa.gov  NaN         bitly
3952   KY  107xZnW  1368815450    1.usa.gov  NaN         bitly
3953  NaN      NaN         NaN          NaN  NaN           NaN
3954   AL  10WWSaQ  1368830835    1.usa.gov  NaN          inws
3955   WI   YmtpnZ  1363711958    1.usa.gov  NaN         bitly
3956   14   YPnFn3  1368833354    1.usa.gov  NaN        hayano
3957  NaN  16n91ZK  1368746683  go.nasa.gov  NaN   nasatwitter
3958   00  10WMBv9  1368826755    1.usa.gov  NaN         bitly

                            ll  nk  \
0     [33.816101, -117.979401]   0
1                          NaN   0
2       [31.5273, -110.360703]   1
3        [29.7633, -95.363297]   1
4                          NaN   0
5        [41.612301, -86.1381]   0
6         [45.007, -92.459099]   1
7                          NaN   0
8       [-33.8615, 151.205505]   0
9                          NaN   0
10       [39.515099, -84.3983]   1
11     [39.131699, -77.288002]   0
12     [37.766602, -84.303101]   1
13    [45.529499, -122.643204]   1
14     [41.760601, -88.320099]   0
15       [29.7633, -95.363297]   0
16     [42.877602, -88.133797]   0
17    [39.802799, -105.087502]   1
18    [44.074402, -121.257401]   1
19     [39.135799, -76.872002]   0
20      [47.606201, -122.3321]   1
21                         NaN   0
22     [44.590698, -91.891197]   0
23                         NaN   0
24                         NaN   0
25                         NaN   0
26                         NaN   0
27                         NaN   0
28                         NaN   0
29                         NaN   0
...                        ...  ..
3929                       NaN   1
3930       [46.242401, 6.1435]   0
3931   [39.089199, -77.183502]   1
3932                       NaN   0
3933   [42.216702, -83.740601]   1
3934   [35.882599, -80.082001]   0
3935    [31.9582, -110.693001]   0
3936  [33.889301, -118.401001]   0
3937                       NaN   1
3938                       NaN   1
3939   [35.854401, 139.968903]   0
3940                       NaN   0
3941     [44.6688, -90.171799]   0
3942  [47.314499, -122.778503]   0
3943       [28.3899, -81.4366]   0
3944   [41.481701, -81.802399]   0
3945   [36.219101, -81.656303]   0
3946                       NaN   1
3947  [41.641201, -111.896599]   0
3948   [35.685001, 139.751404]   1
3949     [37.709, -122.088501]   0
3950   [33.481098, -84.479797]   0
3951     [37.2906, -80.101402]   0
3952   [38.336399, -82.992401]   0
3953                       NaN NaN
3954     [30.657499, -88.1586]   1
3955     [43.060799, -88.1558]   0
3956   [36.083302, 140.116699]   0
3957                       NaN   0
3958     [18.3876, -66.110802]   1

                                                      r           t  \
0                                                direct  1368832205
1                                   http://www.usa.gov/  1368832207
2     http://www.facebook.com/l.php?u=http%3A%2F%2F1...  1368832209
3     http://m.facebook.com/l.php?u=http%3A%2F%2F1.u...  1368832209
4                                http://t.co/CDO9hLTtNT  1368832208
5                                                direct  1368832209
6         http://www.bwsd.k12.wi.us/SitePages/Home.aspx  1368832210
7                                http://t.co/YIsVhFDLj2  1368832211
8                                                direct  1368832213
9                                http://t.co/yPSKO2t5v1  1368832215
10                               https://m.facebook.com  1368832215
11                               http://t.co/u8qVCKx8RK  1368832218
12    http://www.facebook.com/l.php?u=http%3A%2F%2F1...  1368832219
13                               http://t.co/T8EyBbUBJ8  1368832219
14    http://www.z2systems.com/np/clients/kca/news.j...  1368832219
15                               http://t.co/s307mx2qGk  1368832220
16                         http://www.cudahy.k12.wi.us/  1368832220
17                                               direct  1368832220
18                                               direct  1368832222
19                               http://t.co/Dv6Jqbwu8H  1368832222
20                               http://t.co/7K9urpYyc6  1368832224
21                            https://www.facebook.com/  1368832225
22                           http://www.alma.k12.wi.us/  1368832224
23    http://m.facebook.com/l.php?u=http%3A%2F%2F1.u...  1368832224
24                         http://www.wabeno.k12.wi.us/  1368832226
25                          http://addthis.com/hemmings  1368832226
26                          http://addthis.com/hemmings  1368832226
27                          http://addthis.com/hemmings  1368832227
28                          http://addthis.com/hemmings  1368832227
29                          http://addthis.com/hemmings  1368832227
...                                                 ...         ...
3929                             http://t.co/HgiLLFRDtE  1368835778
3930                                             direct  1368835778
3931                                             direct  1368835778
3932                                             direct  1368835780
3933                             http://t.co/orOTdRX5aF  1368835780
3934                                             direct  1368835782
3935                          https://www.facebook.com/  1368835783
3936                                             direct  1368835785
3937                                             direct  1368835786
3938                             http://t.co/q6402O6lFC  1368835786
3939                                             direct  1368835787
3940                                             direct  1368835789
3941                        http://www.colby.k12.wi.us/  1368835791
3942                                 http://fwp.mt.gov/  1368835793
3943  http://www.elnuevodia.com/brillanteexplosionen...  1368835794
3944                                             direct  1368835794
3945  http://www.facebook.com/l.php?u=http%3A%2F%2F1...  1368835793
3946                       http://www.cudahy.k12.wi.us/  1368835795
3947                          https://www.facebook.com/  1368835795
3948                                             direct  1368835795
3949                 http://www.cahighspeedrail.ca.gov/  1368835797
3950                             http://t.co/psBn8njvIB  1368835797
3951  http://www.facebook.com/l.php?u=http%3A%2F%2F1...  1368835798
3952  http://www.facebook.com/l.php?u=http%3A%2F%2F1...  1368835801
3953                                                NaN         NaN
3954  http://m.facebook.com/l.php?u=http%3A%2F%2F1.u...  1368835802
3955                       http://www.cudahy.k12.wi.us/  1368835803
3956                                             direct  1368835804
3957                             http://t.co/XLS75r3BCB  1368835805
3958  http://www.elnuevodia.com/brillanteexplosionen...  1368835806

                        tz                                                  u
0      America/Los_Angeles                                http://www.nsa.gov/
1                           http://answers.usa.gov/system/selfservice.cont...
2          America/Phoenix  http://www.saj.usace.army.mil/Media/NewsReleas...
3          America/Chicago            https://nationalregistry.fmcsa.dot.gov/
4                           http://www.peacecorps.gov/learn/howvol/ab530gr...
5     America/Indianapolis  https://petitions.whitehouse.gov/petition/repe...
6          America/Chicago  http://pld.dpi.wi.gov/files/pld/images/LinkWI.png
7                           http://www.nasa.gov/multimedia/imagegallery/im...
8            Australia/NSW                                http://www.nsa.gov/
9                           http://www.nasa.gov/mission_pages/sunearth/new...
10        America/New_York  http://www.dodlive.mil/index.php/2013/05/the-2...
11        America/New_York  http://doggett.house.gov/index.php/news/571-do...
12        America/New_York  http://www.peacecorps.gov/learn/howvol/ab530gr...
13     America/Los_Angeles   http://www.fws.gov/cno/press/release.cfm?rid=493
14         America/Chicago  http://www.cancer.gov/PublishedContent/Images/...
15         America/Chicago                http://www.army.mil/article/103380/
16         America/Chicago  http://pld.dpi.wi.gov/files/pld/images/LinkWI.png
17          America/Denver  http://www.nws.noaa.gov/com/weatherreadynation...
18     America/Los_Angeles  http://fastlane.dot.gov/2013/05/new-locomotive...
19        America/New_York            http://apod.nasa.gov/apod/ap130517.html
20     America/Los_Angeles  http://www.ice.gov/news/releases/1305/130516sa...
21                          http://www.dodlive.mil/index.php/2013/05/the-2...
22         America/Chicago  http://pld.dpi.wi.gov/files/pld/images/LinkWI.png
23                          http://doggett.house.gov/index.php/news/571-do...
24                          http://pld.dpi.wi.gov/files/pld/images/LinkWI.png
25                          http://answers.usa.gov/system/selfservice.cont...
26                          http://answers.usa.gov/system/selfservice.cont...
27                          http://answers.usa.gov/system/selfservice.cont...
28                          http://answers.usa.gov/system/selfservice.cont...
29                          http://answers.usa.gov/system/selfservice.cont...
...                    ...                                                ...
3929                        http://www.nasa.gov/mission_pages/station/expe...
3930         Europe/Zurich  http://gsaauctions.gov/gsaauctions/aucdsclnk?s...
3931      America/New_York  http://gsaauctions.gov/gsaauctions/aucdsclnk?s...
3932                                                      http://www.nsa.gov/
3933      America/New_York  http://science.nasa.gov/science-news/science-a...
3934      America/New_York                                http://www.nsa.gov/
3935       America/Phoenix  http://cms3.tucsonaz.gov/files/police/media-re...
3936   America/Los_Angeles  http://www.irs.gov/uac/Newsroom/Tax-Relief-for...
3937                        http://www.jpl.nasa.gov/news/news.php?release=...
3938                        http://www.jpl.nasa.gov/news/news.php?release=...
3939            Asia/Tokyo  http://www.doe.gov/articles/energy-department-...
3940                                                      http://www.nsa.gov/
3941       America/Chicago  http://pld.dpi.wi.gov/files/pld/images/LinkWI.png
3942   America/Los_Angeles  http://fwp.mt.gov/hunting/hunterAccess/openFie...
3943      America/New_York  http://science.nasa.gov/media/medialibrary/201...
3944      America/New_York  http://gsaauctions.gov/gsaauctions/aucdsclnk?s...
3945      America/New_York  http://inws.wrh.noaa.gov/weather/alertinfo/103...
3946                        http://pld.dpi.wi.gov/files/pld/images/LinkWI.png
3947        America/Denver  http://doggett.house.gov/index.php/news/571-do...
3948            Asia/Tokyo  http://www.nasa.gov/mission_pages/mer/news/mer...
3949   America/Los_Angeles  http://fastlane.dot.gov/2013/05/new-locomotive...
3950      America/New_York  http://studentaid.ed.gov/repay-loans/understan...
3951      America/New_York  http://doggett.house.gov/index.php/news/571-do...
3952      America/New_York  http://doggett.house.gov/index.php/news/571-do...
3953                   NaN                                                NaN
3954       America/Chicago  http://inws.wrh.noaa.gov/weather/alertinfo/103...
3955       America/Chicago  http://pld.dpi.wi.gov/files/pld/images/LinkWI.png
3956            Asia/Tokyo  http://www.doe.gov/articles/energy-department-...
3957                        http://www.jpl.nasa.gov/news/news.php?release=...
3958   America/Puerto_Rico  http://science.nasa.gov/media/medialibrary/201...

[3959 rows x 18 columns]

In [48]:

In [48]: frame['tz'][:10]
Out[48]:
0     America/Los_Angeles
1
2         America/Phoenix
3         America/Chicago
4
5    America/Indianapolis
6         America/Chicago
7
8           Australia/NSW
9
Name: tz, dtype: object

In [49]: tz_counts = frame['tz'].value_counts()

In [50]: tz_counts[:10]
Out[50]:
America/New_York        903
America/Chicago         686
                        636
America/Los_Angeles     421
America/Puerto_Rico     184
Asia/Tokyo              102
America/Denver           89
Europe/London            85
America/Indianapolis     50
America/Phoenix          40
dtype: int64

In [51]:

In [51]: clean_tz = frame['tz'].fillna("Missing")

In [52]: clean_tz[clean_tz == ""] = "Unknown"

In [53]: tz_counts = clean_tz.value_counts()

In [54]: tz_counts[:10]
Out[54]:
America/New_York        903
America/Chicago         686
Unknown                 636
America/Los_Angeles     421
America/Puerto_Rico     184
Missing                 120
Asia/Tokyo              102
America/Denver           89
Europe/London            85
America/Indianapolis     50
dtype: int64

In [55]: tz_counts[:10].plot(kind='barh',rot=0)
Out[55]: <matplotlib.axes._subplots.AxesSubplot at 0x10d503810>

http://gyazo.com/aeb47e0bfc1155ea5f68569fbaa861ad

In [56]: frame["a"]
Out[56]:
0     Mozilla/5.0 (Linux; U; Android 4.1.2; en-us; H...
1     Mozilla/4.0 (compatible; MSIE 7.0; Windows NT ...
2     Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20...
3     Mozilla/5.0 (Linux; U; Android 4.1.2; en-us; S...
4     Opera/9.80 (Android; Opera Mini/7.5.33286/29.3...
5     Mozilla/5.0 (compatible; MSIE 10.0; Windows NT...
6     Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) G...
7     Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_5 li...
8     Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_3 like ...
9     Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi...
10    Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_3 like ...
11    Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_3 like ...
12    Mozilla/5.0 (iPad; CPU OS 6_1_2 like Mac OS X)...
13    Mozilla/5.0 (Windows NT 5.1; rv:20.0) Gecko/20...
14    Mozilla/4.0 (compatible; MSIE 7.0; Windows NT ...
...
3944    Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; D...
3945    Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) G...
3946    Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.3...
3947    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3)...
3948    Mozilla/5.0 (Linux; Android 4.0.4; SO-03D Buil...
3949    Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) G...
3950    Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_4 like ...
3951    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...
3952    Mozilla/5.0 (iPad; CPU OS 6_1_3 like Mac OS X)...
3953                                                  NaN
3954    Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; M...
3955    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3)...
3956                                   ShortLinkTranslate
3957    Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_4 like ...
3958    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)...
Name: a, Length: 3959, dtype: object

In [57]: frame["a"][30]
Out[57]: u'Mozilla/5.0 (iPod; CPU iPhone OS 6_1_2 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10B146 Safari/8536.25'

In [58]: results = Series([x.split()[0] for x in frame.a.dropna()])

In [59]: results[:5]
Out[59]:
0    Mozilla/5.0
1    Mozilla/4.0
2    Mozilla/5.0
3    Mozilla/5.0
4     Opera/9.80
dtype: object



In [61]: results.value_counts()[:8]
Out[61]:
Mozilla/5.0           3251
Mozilla/4.0            322
CakePHP                 38
ShortLinkTranslate      36
TVersity                30
Opera/9.80              28
Dalvik/1.6.0            19
Xenu                    15
dtype: int64

In [68]: frame
...
[3959 rows x 18 columns]

In [69]: cframe = frame[frame.a.notnull()]

In [70]: cframe
Out[70]:

...

[3839 rows x 18 columns]

In [71]: operating_system = np.where(cframe["a"].str.contains("Windows"), "Windows", "Not Windows")

In [72]: operating_system[:5]
Out[72]:
array(['Not Windows', 'Windows', 'Windows', 'Not Windows', 'Not Windows'],
      dtype='|S11')

In [73]: by_tz_os = cframe.groupby(["tz", operating_system])

In [74]: by_tz_os.size()
Out[74]:
tz
                                Not Windows    484
                                Windows        152
Africa/Cairo                    Windows          3
Africa/Casablanca               Windows          1
Africa/Ceuta                    Not Windows      4
                                Windows          2
Africa/Gaborone                 Windows          1
Africa/Johannesburg             Not Windows      2
America/Anchorage               Not Windows      5
                                Windows          3
America/Argentina/Buenos_Aires  Not Windows      4
                                Windows          7
America/Argentina/Catamarca     Not Windows      1
America/Argentina/Cordoba       Windows          2
America/Asuncion                Windows          1
...
Europe/Sofia       Not Windows    1
Europe/Stockholm   Not Windows    2
                   Windows        2
Europe/Tallinn     Not Windows    1
Europe/Vienna      Not Windows    3
                   Windows        3
Europe/Warsaw      Not Windows    1
                   Windows        1
Europe/Zaporozhye  Windows        1
Europe/Zurich      Not Windows    4
                   Windows        1
Pacific/Auckland   Not Windows    1
                   Windows        8
Pacific/Honolulu   Not Windows    7
                   Windows        5
Length: 170, dtype: int64

In [75]: agg_counts = by_tz_os.size().unstack().fillna(0)

In [76]: agg_counts[:10]
Out[76]:
                                Not Windows  Windows
tz
                                        484      152
Africa/Cairo                              0        3
Africa/Casablanca                         0        1
Africa/Ceuta                              4        2
Africa/Gaborone                           0        1
Africa/Johannesburg                       2        0
America/Anchorage                         5        3
America/Argentina/Buenos_Aires            4        7
America/Argentina/Catamarca               1        0
America/Argentina/Cordoba                 0        2


In [77]: indexer = agg_counts.sum(1).argsort()

In [78]: indexer[:10]
Out[78]:
tz
                                   55
Africa/Cairo                      101
Africa/Casablanca                 100
Africa/Ceuta                       36
Africa/Gaborone                    97
Africa/Johannesburg                42
America/Anchorage                  43
America/Argentina/Buenos_Aires     44
America/Argentina/Catamarca        47
America/Argentina/Cordoba          50
dtype: int64


In [79]: count_subset = agg_counts.take(indexer)[-10:]

In [80]: count_subset
Out[80]:
                      Not Windows  Windows
tz
America/Phoenix                22       18
America/Indianapolis           29       21
Europe/London                  62       23
America/Denver                 41       48
Asia/Tokyo                     88       14
America/Puerto_Rico            93       91
America/Los_Angeles           207      214
                              484      152
America/Chicago               343      343
America/New_York              550      353

In [81]: count_subset.plot(kind="barh", stacked=True)
Out[81]: <matplotlib.axes._subplots.AxesSubplot at 0x10d5095d0>

http://gyazo.com/6aa7c587338d9ca695967a05b5d4baaa

In [82]: normed_subset = count_subset.div(count_subset.sum(1), axis=0)

In [83]: normed_subset.plot(kind="barh", stacked=True)
Out[83]: <matplotlib.axes._subplots.AxesSubplot at 0x112774050>

http://gyazo.com/3035df33d844332cde38c8505d05174d

本の内容と違う。Not Windowsのほうが多い。