Cloud Testing Blog » Server Logs http://www.cloudtesting.com/blog Automated Functional, Cross Browser and Load Testing for Websites Tue, 24 Aug 2010 10:42:17 +0000 en hourly 1 http://wordpress.org/?v=3.0.1 What happens when you post a link on Twitter? http://www.cloudtesting.com/blog/2009/12/23/what-happens-when-you-post-a-link-on-twitter/ http://www.cloudtesting.com/blog/2009/12/23/what-happens-when-you-post-a-link-on-twitter/#comments Wed, 23 Dec 2009 17:33:56 +0000 Phil Smith - Cloud Testing http://www.cloudtesting.com/blog/?p=1249 When a link is put into a Twitter posting, the link gets crawled very quickly by the following agents:

  • @hourlypress
  • Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
  • Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot
  • Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
  • Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)
  • Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
  • User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 OneRiot/1.0 (http://www.oneriot.com)
  • PostRank/2.0 (postrank.com)
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 Me.dium/1.0 (http://me.dium.com)
  • Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3
  • Mozilla/5.0 (compatible; page-store) [email:paul at page-store.com]

Logfiles

This is the result from a web logs perspective (sorry for the formatting):

ec2-174-129-50-201.compute-1.amazonaws.com – - [23/Dec/2009:17:19:36 +0000] “HEAD / HTTP/1.1″ 200 – “-” “@hourlypress”
ec2-75-101-168-49.compute-1.amazonaws.com – - [23/Dec/2009:17:19:36 +0000] “HEAD / HTTP/1.1″ 200 – “-” “@hourlypress”
crawl-66-249-68-131.googlebot.com – - [23/Dec/2009:17:19:37 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
flx1-ppp45.lvdi.net – - [23/Dec/2009:17:19:37 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 OneRiot/1.0 (http://www.oneriot.com)”
64.13.147.189 – - [23/Dec/2009:17:19:37 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)”
jay.favsys.net – - [23/Dec/2009:17:19:39 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot”
65.52.19.122 – - [23/Dec/2009:17:19:39 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)”
89-212-78-95.static.t-2.net – - [23/Dec/2009:17:19:41 +0000] “GET / HTTP/1.1″ 200 – “-” “-”
70.37.66.245 – - [23/Dec/2009:17:19:42 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)”
ec2-174-129-27-223.compute-1.amazonaws.com – - [23/Dec/2009:17:19:42 +0000] “GET /robots.txt HTTP/1.1″ 200 153 “-” “Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)”
ec2-174-129-27-223.compute-1.amazonaws.com – - [23/Dec/2009:17:19:42 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)”
152.201.207.67.svwh.net – - [23/Dec/2009:17:19:45 +0000] “GET / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)”
152.201.207.67.svwh.net – - [23/Dec/2009:17:19:46 +0000] “GET /robots.txt HTTP/1.0″ 200 153 “-” “Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)”
67-220-192-215.hosted.static.webnx.com – - [23/Dec/2009:17:19:47 +0000] “GET / HTTP/1.1″ 200 – “-” “User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)”
flx1-ppp46.lvdi.net – - [23/Dec/2009:17:20:02 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 OneRiot/1.0 (http://www.oneriot.com)”
ec2-174-129-141-109.compute-1.amazonaws.com – - [23/Dec/2009:17:20:04 +0000] “HEAD / HTTP/1.1″ 200 – “-” “PostRank/2.0 (postrank.com)”
ec2-174-129-141-109.compute-1.amazonaws.com – - [23/Dec/2009:17:20:05 +0000] “HEAD / HTTP/1.1″ 200 – “-” “PostRank/2.0 (postrank.com)”
flx1-ppp47.lvdi.net – - [23/Dec/2009:17:20:09 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 Me.dium/1.0 (http://me.dium.com)”
22.e.78ae.static.theplanet.com – - [23/Dec/2009:17:20:20 +0000] “GET /robots.txt HTTP/1.0″ 200 153 “http://portal.cloudtesting.com/” “Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)”
22.e.78ae.static.theplanet.com – - [23/Dec/2009:17:20:26 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)”
ec2-174-129-50-201.compute-1.amazonaws.com – - [23/Dec/2009:17:20:32 +0000] “GET / HTTP/1.1″ 200 – “-” “@hourlypress”
ec2-174-129-50-201.compute-1.amazonaws.com – - [23/Dec/2009:17:20:32 +0000] “GET / HTTP/1.1″ 200 – “-” “-”
ec2-174-129-175-212.compute-1.amazonaws.com – - [23/Dec/2009:17:21:13 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3″
ec2-67-202-51-154.compute-1.amazonaws.com – - [23/Dec/2009:17:23:48 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; page-store) [email:paul at page-store.com]”
ec2-174-129-78-138.compute-1.amazonaws.com – - [23/Dec/2009:17:29:46 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; page-store) [email:paul at page-store.com]”

If you want to see it yourself, just tail your server logs and then paste a URL into Twitter.

]]> http://www.cloudtesting.com/blog/2009/12/23/what-happens-when-you-post-a-link-on-twitter/feed/ 0