Cloud Testing Blog » Twitter http://www.cloudtesting.com/blog Automated Functional, Cross Browser and Load Testing for Websites Tue, 23 Aug 2011 18:41:02 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 What happens when you post a link on Twitter? http://www.cloudtesting.com/blog/2009/12/23/what-happens-when-you-post-a-link-on-twitter/ http://www.cloudtesting.com/blog/2009/12/23/what-happens-when-you-post-a-link-on-twitter/#comments Wed, 23 Dec 2009 17:33:56 +0000 Matt Rees - Cloud Testing http://www.cloudtesting.com/blog/?p=1249 When a link is put into a Twitter posting, the link gets crawled very quickly by the following agents:

  • @hourlypress
  • Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
  • Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot
  • Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
  • Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)
  • Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
  • User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 OneRiot/1.0 (http://www.oneriot.com)
  • PostRank/2.0 (postrank.com)
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 Me.dium/1.0 (http://me.dium.com)
  • Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3
  • Mozilla/5.0 (compatible; page-store) [email:paul at page-store.com]

Logfiles

This is the result from a web logs perspective (sorry for the formatting):

ec2-174-129-50-201.compute-1.amazonaws.com – - [23/Dec/2009:17:19:36 +0000] “HEAD / HTTP/1.1″ 200 – “-” “@hourlypress”
ec2-75-101-168-49.compute-1.amazonaws.com – - [23/Dec/2009:17:19:36 +0000] “HEAD / HTTP/1.1″ 200 – “-” “@hourlypress”
crawl-66-249-68-131.googlebot.com – - [23/Dec/2009:17:19:37 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
flx1-ppp45.lvdi.net – - [23/Dec/2009:17:19:37 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 OneRiot/1.0 (http://www.oneriot.com)”
64.13.147.189 – - [23/Dec/2009:17:19:37 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)”
jay.favsys.net – - [23/Dec/2009:17:19:39 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot”
65.52.19.122 – - [23/Dec/2009:17:19:39 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)”
89-212-78-95.static.t-2.net – - [23/Dec/2009:17:19:41 +0000] “GET / HTTP/1.1″ 200 – “-” “-”
70.37.66.245 – - [23/Dec/2009:17:19:42 +0000] “GET / HTTP/1.1″ 200 20 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)”
ec2-174-129-27-223.compute-1.amazonaws.com – - [23/Dec/2009:17:19:42 +0000] “GET /robots.txt HTTP/1.1″ 200 153 “-” “Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)”
ec2-174-129-27-223.compute-1.amazonaws.com – - [23/Dec/2009:17:19:42 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)”
152.201.207.67.svwh.net – - [23/Dec/2009:17:19:45 +0000] “GET / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)”
152.201.207.67.svwh.net – - [23/Dec/2009:17:19:46 +0000] “GET /robots.txt HTTP/1.0″ 200 153 “-” “Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)”
67-220-192-215.hosted.static.webnx.com – - [23/Dec/2009:17:19:47 +0000] “GET / HTTP/1.1″ 200 – “-” “User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)”
flx1-ppp46.lvdi.net – - [23/Dec/2009:17:20:02 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 OneRiot/1.0 (http://www.oneriot.com)”
ec2-174-129-141-109.compute-1.amazonaws.com – - [23/Dec/2009:17:20:04 +0000] “HEAD / HTTP/1.1″ 200 – “-” “PostRank/2.0 (postrank.com)”
ec2-174-129-141-109.compute-1.amazonaws.com – - [23/Dec/2009:17:20:05 +0000] “HEAD / HTTP/1.1″ 200 – “-” “PostRank/2.0 (postrank.com)”
flx1-ppp47.lvdi.net – - [23/Dec/2009:17:20:09 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0 Me.dium/1.0 (http://me.dium.com)”
22.e.78ae.static.theplanet.com – - [23/Dec/2009:17:20:20 +0000] “GET /robots.txt HTTP/1.0″ 200 153 “http://portal.cloudtesting.com/” “Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)”
22.e.78ae.static.theplanet.com – - [23/Dec/2009:17:20:26 +0000] “HEAD / HTTP/1.0″ 200 – “-” “Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)”
ec2-174-129-50-201.compute-1.amazonaws.com – - [23/Dec/2009:17:20:32 +0000] “GET / HTTP/1.1″ 200 – “-” “@hourlypress”
ec2-174-129-50-201.compute-1.amazonaws.com – - [23/Dec/2009:17:20:32 +0000] “GET / HTTP/1.1″ 200 – “-” “-”
ec2-174-129-175-212.compute-1.amazonaws.com – - [23/Dec/2009:17:21:13 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3″
ec2-67-202-51-154.compute-1.amazonaws.com – - [23/Dec/2009:17:23:48 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; page-store) [email:paul at page-store.com]”
ec2-174-129-78-138.compute-1.amazonaws.com – - [23/Dec/2009:17:29:46 +0000] “GET / HTTP/1.1″ 200 – “-” “Mozilla/5.0 (compatible; page-store) [email:paul at page-store.com]”

If you want to see it yourself, just tail your server logs and then paste a URL into Twitter.

]]> http://www.cloudtesting.com/blog/2009/12/23/what-happens-when-you-post-a-link-on-twitter/feed/ 0 New Cloud Testing Mailing list http://www.cloudtesting.com/blog/2009/05/29/new-cloud-testing-mailing-list/ http://www.cloudtesting.com/blog/2009/05/29/new-cloud-testing-mailing-list/#comments Fri, 29 May 2009 10:34:15 +0000 cloudtesting http://blog.cloudtesting.com/?p=98 You can now sign up for the Cloud Testing Mailing List. You will be able to keep up to date with all announcements and updates from us.

You can of course also follow us on Twitter at twitter.com/cloudtesting, or subscribe to the RSS feed of our Blog at blog.cloudtesting.com.

]]> http://www.cloudtesting.com/blog/2009/05/29/new-cloud-testing-mailing-list/feed/ 0