Great news out of Tweetmeme and Techcrunch: Cligs is the fourth most popular URL shortener in use on Twitter, and the 5th most used domain in links overall.

The graph below, says it all:

Thank you to everyone using Cligs, on Twitter and elsewhere! This is a great day for the whole community!

Sources: Tweetmeme (live stats…) and Techcrunch.

I’ve talked many times about the New Cligs System, this mythical entity that will be a much better version of Cligs that will fix all the downtime and bring a 1000 years of world peace.

Well it’s almost here! As I write this, the first data from the current system is being imported into the new system. The actual launch won’t be for a few more weeks, but it’s definitely started.

So what’s the moving plan? It’s like this:

• Finish the code and freeze development of new features. I have a shrinking to-do list and I’m working through it at a good pace.
• Import as much data as possible into the new system. The key bits of data to move now are:
  • User account data, so you can log in;
  • All of the cligs and their Right Clig (geotargeting) rules, so that the new system knows how to forward;
  • API keys.
• Switch the DNS so that the new system takes over from the current system. This switch will take a few days (there are tricks to hopefully make that go faster). During this time both systems will be working and there will be a very geeky way of keeping them in synch!
• Once the current system is no longer receiving requests, the analytics data will be moved as it’s a huge data set and it will need some processing to make it work fully in the new system. Yes, the analytics are much improved and so the current data needs to be transformed.
• When everything is moved and everything is sorted, the current system will be shut down.

Thoughts? Comments? Feedback? Comment below or email me.

Given the massive spike in traffic recently, I just back-ported a significant code change from the new Cligs system to the current one. The change means that the server will forward cligs much (much!) more quickly while at the same time lower system resources. The upshot is hopefully a more stable service handling more traffic.

As this is a significant change, I would like you to be vigilant for any bugs or oddities please. I have been testing it all morning and it’s worked beautifully in the tests, but you never know. In particular, the biggest potential source of bugs is when editing cligs, although that got the most testing and came up clean.

The new system is approaching feature partiy (well, in some areas it is already a lot better than the current one) and will hopefully be finished soon.

For the technically inclined: What’s the update? A much better caching layer that blows the old one out of the water.

As many have noticed, Cligs was not responding to most requests over the past 6 hours. The reasons why this happened is actually a rolling series of problems that cascaded into each other.

Fundamentally, the starting cause was another service hitting the API at a very high rate, on top of the normal traffic. A couple of hours after that started happening, a malicious bot started hitting Cligs requesting random URLs, generating a lot of 404 pages and basically doing nothing useful as far as I can tell. At least this bot was kind to limit itself to just 1 request a second, but it couldn’t have come at a worse time.

So I blocked both, reconfigured MySQL and Apache (both of which were straining under the load) and now Cligs is back up again.

This is a temporary fix as I redesign some fundamental things about Cligs to make it more resilient. What the past few hours have exposed is a weakness in the code and architecture that need to be addressed.