That was much quicker than I expected, mostly due to my very helpful friend Michael. Without getting all technically geeky on you, Michael showed me a very neat solution to pull out the bot data in an efficient way. The efficiency means it’s now 100x faster than it used and it uses less memory.

End result? The bot analytics are back and the Clig hardware lives to die some other day Thanks Michael!

Following yesterday’s post about Cligs slowness, it’s clear that trimming the latest clig hits count from 100 to 10 helped a lot. The service is still occasionally acting up, so now I’ve disabled the bot analytics.

This is temporary! The bot analytics are being re-built as part of the changes I described yesterday to make them less draining on the server. I’m hoping everything will be done by early next week.

Speaking of which, the backend changes are half done so I’m slightly ahead of the schedule I’ve put for it - yeah, that happens sometimes! Fingers crossed the second half works out well too!

Cligs has been a bit slower than usual which means that we’ve found the next growth bottleneck that needs fixing. I’d like to explain to you what’s going on and tell you how it’s going to be fixed.

The Problem

The biggest problem is the My Cligs page, specifically, the "Cligs Getting Last 100 Hits" feature. I had reports from someone whose My Cligs page takes a full 45 seconds to load, and that’s not that unusual. All that time, the Cligs server is building the page, which slows down the service. At peak hours when everyone is checking their stats, Cligs can get really slow.

The second biggest problem is the introduction of the new bot detection analytics. This feature makes Cligs much slower than it should be, and I was not able to foresee that before launch. Now that it’s launched and everyone is commenting on how cool it is (i.e. using it!), Cligs is starting to strain.

The Fixes

Firstly, as a temporary measure, I’ve tweaked the "Cligs Getting Last 100 Hits" to make it list only the last 10 cligs that got hits. This makes the page loading much faster and doesn’t strain the server.

In the next few days, I’ll be tweaking the database and changing the way it stores data. This has two side effects:

• It will make the analytics run much faster.
• It will enable me to deploy new types of analytics and build new features that I’ve always wanted to add. One of those features will be unique to Cligs just like the Right Clig geotargeting.

When? The backend is being changed tonight and tomorrow, and I’m hoping to make the switch over the weekend quiet period just in case things go into digital lala land. The new features will be built once the changes are confirmed as working.

Summary

This post documents an analysis of 10.2 million tweets containing 2 million links. Those links were analyzed to understand which domains are most used. The analysis shows that:

• tinyurl.com is the most used domain name;
• cli.gs is the 18th most used domain name;
• Of the top 50 domains used in links, 18 are URL shortening services;
• Of the URL shortening services, Cligs is 10th most used.

As the owner of a URL shortening service here on Cligs, I’m very interested in the Twitter market, hence this analysis. I’m sharing the results because they are very interesting in their own right.

Note: A list of references is found at the end of this post.

Introduction

On December 22, a massive Twitter data scrape was released, with the blessing of Twitter, no less. How massive? Let’s look at some of the numbers:

• 10.2 million tweets…
• …from 8 million users…
• …containing 2 million links…
• …and 219 thousand hashtags.

And a lot more.

Of particular interest are the links data, handily extracted into a separate file. The file contained data for 2071291 links. These approximately 2.1 million links were analyzed to extract the domain name of the URL of the link. For example, for a link URL for http://cli.gs/abc123, the domain name is cli.gs.

The analysis counted the occurances of the domain names in the 2.1 million links. The domain names were then sorted by their occurance count in descending order. The top 50 domains were analyzed in detail.

Top 50 Domains Used on Twitter

First the data, which you can download as a PDF file at the end:

Analysis of Top URL Shortners Used on Twitter

The top 18 URL shortners accounted for 1395892 links, or 67.4% of them. Some URL shortening services have multiple alternative domains their users can opt to use. If we group these sister domains and look at the URL shortners, the data looks as follows:

Or in a more graphical form:

Thoughts & Conclusions

• We have an absolute classic of a long-tail data set: The top 50 domains accounted for 1628666 links (78.6%). This is a very narrow head and a very long tail of domain usage.
• The tinyurl.com domain dominates, accounting for 75% of all URL shortening on Twitter. Thoughts about how much valuable traffic Twitter is simply throwing away by not owning on a postcard please.
• Cligs is the 10th most used on Twitter which is not bad for a one-man show in 3 months!
• Look at ff.im. That’s Friendfeed’s posting to Twitter mechanism. That’s a lot of posting going on there!
• Twitpic is much more popular than I thought it would be. It’s the top non-shortening service used in links, and number 3 overall in the link rankings. I already had great respect for the Twitpic guy (yes, it’s also a one-man show) and now we have numbers to show just how great the service is.
• Sanity check: I watch Twitter with a very keen eye, and my gut feeling is that the usage data above actually feels right. Nothing in it is very surprising and a lot of the rankings can be explained fairly easily. It’s important in any data analysis to do this kind of sanity check and I urge you to look at the numbers again and see if they make sense to you too.

References

• Data source: get.theinfo Google Group.
• Top 50 Domains (PDF file download): Top 50 Domains used in Twitter links.

As of just now, Clig Details pages got a new feature: analytics showing you bot traffic.

What’s a bot? A bot/robot/crawler/spider is software that follows links. There are good bots, like GoogleBot which indexes the web to build Google’s search index, and there are bad bots that scrape content and do other nasties.

Since launching Cligs, the analytics showed you all traffic clumped together without breaking it down into humans clicking through versus bots automatically requesting the clig. Today’s update is a basic start to building more on this kind of breakdown and this type of analytics.

So what does today’s update do? Go to any Clig Details page (the bar graph icon) and under the Total Hits section you’ll see the breakdown; for a recent clig, this is what I see:

As you can see, it’s a very simple breakdown: total number of hits (as previously was shown) with two items underneath it showing the number of hits that are bots and those that are humans.

So how does Cligs detects bots? Well some of them are very easy to detect and some not. Given the millions of clig requests Cligs has seen since launching, finding bots is easier than usual: there is a lot more data to mine for interesting traffic patterns.

The traffic analysis I did indentified a lot of IP addresses that exhibit bot-like behavior. I manually checked the top 100 IP addresses (yes, manually) and confirmed they are bots. These IP addresses were then added to a special list that the Cligs analytics check to produce the breakdown.

This technique has one very important side-effect: not all bots will be detected so the number of bots Cligs gives you is the minimum guaranteed number of hits that are bots. It may be that the rest of the traffic contains bot traffic too that’s not detected. No technique can guarantee 100% detection of human vs bot traffic but we can have a honest crack at it.

With time, more IP addresses will be added to detect more bots. Also, Cligs will soon be able to show you which bots have generated traffic on your cligs. Eventually, I’d like to see how to best merge the Latest Search Engine Bot Sightings section with the detailed bot analytics as search engine bots are one type of bots. Ideas and thoughts welcome