Google Bot and yahoo bot IP's

Ph1ngering · by **Ph1ngering** » Mon Feb 25, 2008 8:01 pm

Just curious if any one has a good list of what IPs google and yahoo uses for there crawls..

I have one crawling my site 24x7 its seems, but i don't think its the deep crawl bot, its the daily fresh bot.. Just curious if any one knows the different ip ranges between the fresh bot and the deep bot.. (for lack of better terms)

66.249.73.57 google and 74.6.28.42 yahoo are both crawling me now..

Thanks
--Craig

Google ip's

66.249.65.133
66.249.67.72
66.249.72.137

Yahoo ip's
72.30.179.200
72.30.179.204
72.30.179.212
72.30.179.215
72.30.179.217
72.30.179.218
72.30.179.225
72.30.179.230
72.30.179.233
72.30.179.235
74.6.19.107
74.6.19.109
74.6.19.141
74.6.19.142
74.6.19.151
74.6.19.154
74.6.19.159
74.6.19.160
74.6.19.165
74.6.19.198
74.6.19.200
74.6.19.202
74.6.19.210
74.6.19.214
74.6.19.218
74.6.19.221
74.6.19.222
74.6.19.223
74.6.19.232
74.6.19.233
74.6.19.236
74.6.19.237
74.6.19.75
74.6.19.80
74.6.19.82
74.6.20.101
74.6.20.103
74.6.20.114
74.6.20.116
74.6.20.12
74.6.20.136
74.6.20.14
74.6.20.147
74.6.20.153
74.6.20.16
74.6.20.161
74.6.20.166
74.6.20.17
74.6.20.19
74.6.20.198
74.6.20.203
74.6.20.207
74.6.20.21
74.6.20.211
74.6.20.212
74.6.20.214
74.6.20.219
74.6.20.222
74.6.20.224
74.6.20.229
74.6.20.23
74.6.20.26
74.6.20.32
74.6.20.36
74.6.20.39
74.6.20.41
74.6.20.70
74.6.20.71
74.6.20.75
74.6.22.13
74.6.22.15
74.6.22.18
74.6.22.19
74.6.22.205
74.6.22.21
74.6.22.211
74.6.22.22
74.6.22.223
74.6.22.224
74.6.22.225
74.6.22.226
74.6.22.229
74.6.22.230
74.6.22.233
74.6.22.31
74.6.22.37
74.6.22.39
74.6.22.41
74.6.22.44
74.6.22.49
74.6.23.105
74.6.23.108
74.6.23.110
74.6.23.117
74.6.23.121
74.6.23.122
74.6.23.139
74.6.23.142
74.6.23.147
74.6.23.15
74.6.23.150
74.6.23.159
74.6.23.164
74.6.23.199
74.6.23.203
74.6.23.204
74.6.23.210
74.6.23.213
74.6.23.216
74.6.23.22
74.6.23.220
74.6.23.221
74.6.23.222
74.6.23.231
74.6.23.233
74.6.23.236
74.6.23.26
74.6.23.29
74.6.23.38
74.6.23.70
74.6.23.82
74.6.24.10
74.6.24.102
74.6.24.104
74.6.24.105
74.6.24.107
74.6.24.112
74.6.24.123
74.6.24.13
74.6.24.134
74.6.24.137
74.6.24.140
74.6.24.142
74.6.24.145
74.6.24.146
74.6.24.152
74.6.24.153
74.6.24.169
74.6.24.18
74.6.24.198
74.6.24.20
74.6.24.203
74.6.24.209
74.6.24.210
74.6.24.214
74.6.24.217
74.6.24.219
74.6.24.22
74.6.24.221
74.6.24.222
74.6.24.223
74.6.24.226
74.6.24.234
74.6.24.26
74.6.24.28
74.6.24.30
74.6.24.34
74.6.24.36
74.6.24.37
74.6.24.45
74.6.24.46
74.6.24.70
74.6.24.72
74.6.24.82
74.6.25.103
74.6.25.104
74.6.25.117
74.6.25.118
74.6.25.120
74.6.25.123
74.6.25.124
74.6.25.137
74.6.25.139
74.6.25.143
74.6.25.147
74.6.25.148
74.6.25.15
74.6.25.160
74.6.25.161
74.6.25.172
74.6.25.19
74.6.25.199
74.6.25.200
74.6.25.205
74.6.25.209
74.6.25.211
74.6.25.216
74.6.25.217
74.6.25.220
74.6.25.223
74.6.25.24
74.6.25.31
74.6.25.32
74.6.25.36
74.6.25.70
74.6.25.74
74.6.25.75
74.6.25.77
74.6.25.81
74.6.25.82
74.6.26.103
74.6.26.105
74.6.26.114
74.6.26.137
74.6.26.139
74.6.26.140
74.6.26.143
74.6.26.144
74.6.26.145
74.6.26.154
74.6.26.156
74.6.26.16
74.6.26.160
74.6.26.163
74.6.26.165
74.6.26.166
74.6.26.199
74.6.26.207
74.6.26.220
74.6.26.221
74.6.26.228
74.6.26.231
74.6.26.232
74.6.26.31
74.6.26.38
74.6.26.39
74.6.26.72
74.6.26.82
74.6.27.103
74.6.27.123
74.6.27.14
74.6.27.16
74.6.27.31
74.6.27.34
74.6.27.35
74.6.27.47
74.6.27.72
74.6.28.101
74.6.28.109
74.6.28.11
74.6.28.121
74.6.28.134
74.6.28.138
74.6.28.139
74.6.28.149
74.6.28.150
74.6.28.160
74.6.28.162
74.6.28.163
74.6.28.164
74.6.28.170
74.6.28.173
74.6.28.18
74.6.28.200
74.6.28.206
74.6.28.21
74.6.28.212
74.6.28.214
74.6.28.215
74.6.28.216
74.6.28.217
74.6.28.218
74.6.28.22
74.6.28.221
74.6.28.224
74.6.28.229
74.6.28.237
74.6.28.25
74.6.28.31
74.6.28.36
74.6.28.39
74.6.28.47
74.6.28.73
74.6.28.76
74.6.28.78
74.6.29.13
74.6.29.30
74.6.29.37
74.6.29.38
74.6.29.40
74.6.29.42
74.6.29.43
74.6.29.49
74.6.7.101
74.6.7.112
74.6.7.122
74.6.7.124
74.6.7.126
74.6.7.135
74.6.7.138
74.6.7.139
74.6.7.140
74.6.7.146
74.6.7.151
74.6.7.154
74.6.7.166
74.6.7.169
74.6.7.173
74.6.7.71
74.6.7.74
74.6.7.81

Ph1ngering · by **Ph1ngering** » Wed Feb 27, 2008 1:01 am

nice list, but do you know which ones are the daily quick crawl and which is the deep crawl?

Peter77 · by **Peter77** » Thu Feb 28, 2008 2:32 am

My guess is the quick crawler... I might be wrong, but I am not 100% for sure.
Each day those are the ip's that show up on my stats and the ip's have been marked to have visisted 1 to 4 times in one 24 hour day.

dcz · by **dcz** » Sat Mar 01, 2008 4:37 pm

Ph1ngering wrote:nice list, but do you know which ones are the daily quick crawl and which is the deep crawl?

I'm not convinced that such distinction exist for real. What made you think that some ip where only involved into quick or deep crawl and that this could have any incidence ?

++

Ph1ngering · by **Ph1ngering** » Mon Mar 10, 2008 1:29 am

Just something I had saw some where on the internet.. But it has said that monthly google will deeply crawl your site vrs just looking for one off updates.

Which would make sense to me to do it this way and would explain why it takes things that were index'd months ago so long to be updated again.

dcz · by **dcz** » Wed Mar 12, 2008 2:10 pm

Well, this is for sure, but it's rather related to the way Google determine and schedules crawling, I don't think that there is a distinction in between several bots to do so. And it would not make a difference in case it would, since it all goes down to a crawling list with priorities.

++

vardis · by **vardis** » Sun Nov 23, 2008 8:58 pm

Great info - nice one.

notouch2008 · by **notouch2008** » Tue Aug 18, 2009 5:56 pm

Thanks for sharing.

Typo · by **Typo** » Tue Aug 18, 2009 10:28 pm

Keep in mind that list is already 6 months old and could be outdated or incomplete now.

Typo

dcz · by **dcz** » Sat Aug 22, 2009 11:50 am

Google even provided with a smarter and always up to date way to verify googlebot : http://googlewebmastercentral.blogspot. ... lebot.html

Telling webmasters to use DNS to verify on a case-by-case basis seems like the best way to go. I think the recommended technique would be to do a reverse DNS lookup, verify that the name is in the googlebot.com domain, and then do a corresponding forward DNS->IP lookup using that googlebot.com name; eg:

> host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

> host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

I don't think just doing a reverse DNS lookup is sufficient, because a spoofer could set up reverse DNS to point to crawl-a-b-c-d.googlebot.com.

Of course, this is still not fully safe for those willing to go for cloaking, since google could still verify your site with other hosts / ip, but at least, it does not requires you to keep your ip list up to date.

++

Google Bot and yahoo bot IP's

Google Bot and yahoo bot IP's

Advertisement

Re: Google Bot and yahoo bot IP's

Re: Google Bot and yahoo bot IP's

Re: Google Bot and yahoo bot IP's

Who is online

Advertisement