What happens when you post a link on Twitter?
When a link is put into a Twitter posting, the link gets crawled very quickly by the following agents:
- @hourlypress
- Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
- Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot
- Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
- Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)
- Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
- User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)
- Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 OneRiot/1.0 (http://www.oneriot.com)
- PostRank/2.0 (postrank.com)
- Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 Me.dium/1.0 (http://me.dium.com)
- Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
- Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3
- Mozilla/5.0 (compatible; page-store) [email:paul at page-store.com]
Logfiles
This is the result from a web logs perspective (sorry for the formatting):
ec2-174-129-50-201.compute-1.amazonaws.com – - [23/Dec/2009:17:19:36 +0000] “HEAD / HTTP/1.1″ 200 – “-” “@hourlypress”
ec2-75-101-168-49.compute-1.amazonaws.com – - [23/Dec/2009:17:19:36 +0000] “HEAD / HTTP/1.1″ 200 – “-” “@hourlypress”
crawl-66-249-68-131.googlebot.com – - [23/Dec/2009:17:19:37 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
flx1-ppp45.lvdi.net – - [23/Dec/2009:17:19:37 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 OneRiot/1.0 (http://www.oneriot.com)”
64.13.147.189 – - [23/Dec/2009:17:19:37 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)”
jay.favsys.net – - [23/Dec/2009:17:19:39 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot”
65.52.19.122 – - [23/Dec/2009:17:19:39 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)”
89-212-78-95.static.t-2.net – - [23/Dec/2009:17:19:41 +0000] “GET / HTTP/1.1″ 200 – “-” “-”
70.37.66.245 – - [23/Dec/2009:17:19:42 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)”
ec2-174-129-27-223.compute-1.amazonaws.com – - [23/Dec/2009:17:19:42 +0000] “GET /robots.txt HTTP/1.1″ 200 153 “-” “Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)”
ec2-174-129-27-223.compute-1.amazonaws.com – - [23/Dec/2009:17:19:42 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)”
152.201.207.67.svwh.net – - [23/Dec/2009:17:19:45 +0000] “GET / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)”
152.201.207.67.svwh.net – - [23/Dec/2009:17:19:46 +0000] “GET /robots.txt HTTP/1.0″ 200 153 “-” “Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)”
67-220-192-215.hosted.static.webnx.com – - [23/Dec/2009:17:19:47 +0000] “GET / HTTP/1.1″ 200 – “-” “User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)”
flx1-ppp46.lvdi.net – - [23/Dec/2009:17:20:02 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 OneRiot/1.0 (http://www.oneriot.com)”
ec2-174-129-141-109.compute-1.amazonaws.com – - [23/Dec/2009:17:20:04 +0000] “HEAD / HTTP/1.1″ 200 – “-” “PostRank/2.0 (postrank.com)”
ec2-174-129-141-109.compute-1.amazonaws.com – - [23/Dec/2009:17:20:05 +0000] “HEAD / HTTP/1.1″ 200 – “-” “PostRank/2.0 (postrank.com)”
flx1-ppp47.lvdi.net – - [23/Dec/2009:17:20:09 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 Me.dium/1.0 (http://me.dium.com)”
22.e.78ae.static.theplanet.com – - [23/Dec/2009:17:20:20 +0000] “GET /robots.txt HTTP/1.0″ 200 153 “http://portal.cloudtesting.com/” “Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)”
22.e.78ae.static.theplanet.com – - [23/Dec/2009:17:20:26 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)”
ec2-174-129-50-201.compute-1.amazonaws.com – - [23/Dec/2009:17:20:32 +0000] “GET / HTTP/1.1″ 200 – “-” “@hourlypress”
ec2-174-129-50-201.compute-1.amazonaws.com – - [23/Dec/2009:17:20:32 +0000] “GET / HTTP/1.1″ 200 – “-” “-”
ec2-174-129-175-212.compute-1.amazonaws.com – - [23/Dec/2009:17:21:13 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3″
ec2-67-202-51-154.compute-1.amazonaws.com – - [23/Dec/2009:17:23:48 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; page-store) [email:paul at page-store.com]”
ec2-174-129-78-138.compute-1.amazonaws.com – - [23/Dec/2009:17:29:46 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; page-store) [email:paul at page-store.com]”
If you want to see it yourself, just tail your server logs and then paste a URL into Twitter.
Did you enjoy this post? Why not subscribe to our feed and get articles like this delivered automatically to your feed reader.
Comments
No comments yet.
Sorry, the comment form is closed at this time.