A few users have been reporting problems with logins (still!), and after a lot of troubleshooting, I narrowed it down to some kind of issue between Cligs and Internet Explorer 8 and Cligs and Safari. Firefox and Chrome were always happy to let users login normally, but IE8 and Safari didn’t.

It took me a long time to figure out the problem as I wasn’t able to reproduce it until earlier today. I’m happy to report that all my test cases of failed logins are now fixed, so I’m confident this will apply widely to all affected users.

Apologies for the delay on this one; the slowness to reply was due to my inability to see the problem first hand.

To boot, the fix makes some Cligs run much faster. Not a bad start to the week!

A small minority of users have had problems logging in, and a lot of people had their logins being constantly dropped. The former is being fixed as we speak and the latter is fixed now.

For the dropped logins: Cligs required you to regularly login even if you had just logged in minutes ago. This was a silly default setting in PHP which I corrected. You may need to clear your cookies for this to work, as browsers can get confused with multiple logins like this.

For the account problems: as I said on Twitter, some user accounts did not transfer correctly to the new server. I figured out why and now all accounts are being rei-mported. I expect all of them to be completely imported in the next 2-3 hours, but most likely it will be sooner - it’s just going through the accounts one by one and so your account is very likely to have already been imported. Worth a check.

Again, please clear your cookies! Thanks!

The new server is doing really well, and so I’ve switched back on the real-time analytics without any problems. I’ll keep watching it, but I’m very happy with how it’s going.

Next up is the logins issue.

Following last night’s post, I implemented a fix for what I thought was the problem. It most definitely helped, but the service is not running at 100% speed. It’s certainly responding much more quickly. Cligs is usable albeit it’s slow - yesterday it was nowhere near usable.

A very important note: The analytics are being logged but may not show up in your analytics reports for a few hours. This time frame is temporary and will come down to minutes/seconds very soon. The delay is a side-effect of how the fix worked - it makes a trade-off between really live and slow and slightly delayed and fast/stable.

Rest assured I’ll be working hard to bring Cligs back its full glory for you!

In a nutshell: the service moved and the server is very slow. Login problems are the most common symptom. I think I know why it’s slow, but won’t know for sure till I modify some code deep in the innards of the Cligs application. This will happen tomorrow (it’s past midnight here in the UK).

Long story for the geek minded: The new service runs off nginx with FastCGI to run the PHP code of Cligs. The problem is that the FastCGI processes are collectively taking up 100% of the CPU, occasionally dipping to only 70%.

The problem seems to be in writing at a very high rate to SimpleDB; the only way to fix this is to rebuild the live analytics saving code.

Just to re-iterate a very important point that people are asking about: the analytics of your cligs have NOT moved yet and will move over the next few days. Don’t worry they have not been deleted and will not be deleted.

Cligs created on the new system will have the full analytics as expected.

Just triggered the move. Full details in the post from last night.

Everything is going according to plan, but with a few hours delay. The move will be triggered later tonight or first thing tomorrow (Monday) morning.

Progress so far:

• All URLs up to the day of the hack have been moved, including restoring the old hacked ones.
• As I type this, I’m moving the cligs created in the past week since the hack.

The move will occur as follows: I’ll modify the DNS entry for the domain cli.gs to point it to the new IP address. This, as we painfully learned in April, is mostly complete within 24 hours, but some DNS caches will take over a week to complete. Don’t worry, as there is plan in action to ensure there is no disruption of service.

After the move completes, the task of moving over the full analytics data will commence. This, again, should take a few days. I’ll start it from the most recent data and work backwards to ensure that your most recent hits are ported early in the process.

Help please: The new service is much better and brings a significant change in the user interface. And since this is a move of a complicated application with lots of data, no doubt there are things that will need polishing. A number of beta testers have given hugely valuable feedback already, but no doubt the community collectively will spot even more things to tweak.

When this move is completed, the fun starts again: I’ve been working on new analytics that I’d like to roll out, and they will come out over the next few weeks and months.

As ever, any questions or feedback, please contact me. I may be a bit slower than usual these days, but I’ll get to you in due course.

The data restoration is going very smoothly now. I did find a way to make it go a lot faster earlier, but hit what appeared to be rate limits between EC2 and SimpleDB (the new Amazon infrastructure). I’ve slowed it down a bit and it’s been working very well.

At this rate, I anticipate that the move to the new servers will occur over the weekend; this is the upper-end of the estimate I gave earlier.

A little update and more details and answers to questions:

Most important thing: your passwords are safe. They are stored encrypted and were never at risk in this attack.

What’s going on: I’m using this opportunity to move to the new Cligs system as it’s pretty much ready. As we speak and since last night, the data is being imported into the new database. It’s going well, but is slower than we want. Assuming all goes well, I’m guessing we’ll be ready by tomorrow at some point, but could take longer. Remember that’s moving millions of URLs. Once the URLs are moved, a DNS change will be triggered along with a live-sync system (that’s another post) to move to the new platform.

All told, we’ll be on the brand new system with all the data by the weekend. Also, the upshot is that the cleaned up redirects will not be visible until the new system rolls out.

More numbers because, well, we like analytics like this (it’s kinda the purpose of Cligs):

• 2,188,978 URLs were edited in the attack (that’s the 2.2 million number I quoted earlier).
• Of those, 2,180,484 have not been deleted or disabled. (Remember when Cligs deletes a clig, it leaves a stub that remembers the actual clig so that it’s not used again in the future.)
• Of those, only 161,232 are not in the backup. That’s only 7% of the affected URLs. Not bad all things considered. Users will be able to edit those eventually and I’m happy to work with users wishing to do bulk updates to their accounts; details of this process will come after the move completes (one step at a time).
• These 161 thousand, 95,123 (59%) are not associated with a specific user account. Those are really lost forever, and will change them to redirect to the Cligs home page.
• Of the 161 thousand that are associated with user accounts (i.e., the remaining 41%), only 2,268 user accounts are affected. This represents a small percentage of the total number of active accounts. Most accounts have a tiny number of cligs affected (the distribution is a classic long tail).

I’ll keep you updated as things progress.