管理しているサーバーに Mozilla/5.0 (compatible; crawler) というUserAgentを名乗るアクセスが超大量にあるため、情報共有のために超久しぶりに記事を作成しました。
UserAgent
Mozilla/5.0 (compatible; crawler)
IPアドレス
全て、HOSTが ○○.ap-southeast-1.compute.amazonaws.com に所属しているIPアドレスのようです。
3.0.36.93
3.0.37.233
3.0.45.179
3.0.47.233
3.0.150.19
3.0.166.8
3.0.208.184
3.1.28.195
3.1.35.115
3.1.41.121
3.1.55.166
3.1.140.69
3.1.156.23
3.1.175.59
3.1.176.125
3.1.232.118
13.213.84.21
13.213.93.162
13.213.127.104
13.213.157.13
13.213.171.53
13.213.177.145
13.213.193.94
13.213.209.156
13.213.233.255
13.214.0.180
13.214.22.82
13.214.103.223
13.214.116.108
13.214.244.23
13.215.52.199
13.215.112.129
13.215.124.152
13.215.168.159
13.215.188.31
13.215.229.29
13.215.231.211
13.228.74.91
13.228.88.208
13.228.118.185
13.228.137.154
13.228.227.67
13.229.12.223
13.229.12.225
13.229.16.251
13.229.27.234
13.229.31.250
13.229.33.21
13.229.165.132
13.250.87.217
13.250.138.232
13.250.142.18
13.250.202.73
13.250.203.189
13.250.227.154
13.250.252.231
13.251.39.246
13.251.50.26
13.251.83.43
13.251.147.94
13.251.241.103
18.136.0.106
18.136.10.12
18.136.16.225
18.136.36.22
18.136.83.96
18.136.115.59
18.136.128.0
18.136.160.164
18.136.192.146
18.136.246.89
18.138.13.230
18.138.67.243
18.138.77.105
18.138.91.172
18.138.160.105
18.138.162.156
18.138.168.48
18.138.206.198
18.139.6.74
18.139.19.113
18.139.55.40
18.139.61.118
18.139.77.112
18.139.81.237
18.140.93.125
18.140.103.95
18.140.106.216
18.140.141.71
18.140.152.11
18.140.180.181
18.141.87.4
18.141.118.94
18.141.136.97
18.141.242.15
18.141.253.176
18.142.25.163
18.142.89.160
18.142.97.33
18.142.153.169
18.142.166.146
18.142.191.196
18.142.233.159
18.143.56.174
18.143.72.152
18.143.82.121
18.143.128.13
18.143.245.207
46.137.206.146
46.137.211.165
46.137.229.112
46.137.241.255
47.128.161.133
47.128.189.246
47.130.17.238
47.130.30.119
47.130.41.114
47.130.45.182
47.130.48.168
47.130.57.161
47.130.61.192
47.130.61.219
47.130.69.52
47.130.76.37
47.130.79.23
47.130.80.38
47.130.86.31
52.74.58.47
52.74.82.10
52.74.129.116
52.74.142.177
52.74.162.53
52.74.177.159
52.74.185.192
52.74.204.10
52.74.206.17
52.74.247.162
52.74.253.55
52.76.105.201
52.76.111.236
52.76.247.228
52.77.43.229
52.77.55.177
52.77.143.193
52.77.167.177
52.77.171.255
52.77.190.52
52.77.201.81
52.220.43.33
52.220.55.39
52.220.69.218
52.220.91.113
52.220.99.112
52.220.106.52
52.220.113.184
52.220.174.74
52.220.206.48
52.220.240.16
52.221.7.10
52.221.20.215
52.221.80.233
52.221.86.130
52.221.86.223
52.221.108.21
52.221.140.211
54.151.168.53
54.151.185.30
54.151.249.108
54.151.255.242
54.169.18.221
54.169.131.194
54.169.160.27
54.169.181.17
54.169.227.4
54.179.3.9
54.179.63.7
54.179.79.203
54.179.95.236
54.179.108.227
54.251.77.221
54.251.100.82
54.251.125.148
54.251.197.106
54.251.227.252
54.251.253.210
54.251.255.21
54.254.5.110
54.254.24.54
54.254.35.214
54.254.37.200
54.254.54.29
54.254.108.206
54.254.110.64
54.254.148.114
54.254.148.222
54.254.208.62
54.254.218.15
54.255.26.49
54.255.66.232
54.255.157.237
54.255.218.6
122.248.208.194
122.248.249.153
175.41.150.136
拒否の方法
以下の記事を参考にしてください。