This is a fairly techie tale of a company CEO that was a little sloppy (well, not paranoid enough), a couple of security holes in two different systems and a lot of patient work to take over a website. In short, a small group of hackers managed to wrest control of the Google Apps account and voicemail box of a hosting company president, then used that information to temporarily take over a group of websites hosted by that company. The attack was detected and stopped quickly and control was returned to the hosting company, but it took a lot of work and a lot of people to thwart one attack.
A detailed blog post by the CEO of the hosting company, CloudFlare's Matthew Prince, tells the entire story, including mistakes and apologies. CloudFlare even went so far as to create a nice timeline to illustrate the timing of the hack, the response, and the key details. Going one step even further, CloudFlare took the opportunity to use this incident as a teachable moment, to inform and advise their clients and the general public in how to protect one's self from the same abusive tactics. Even though CloudFlare made some mistakes, they more than made up for it in their quick and thorough response and then took it one step farther to help others to prevent the same attack. This is the kind of excellence in community building and collaboration that warms my heart and confirms my faith in our society and people in general. Well done.
Google fixed their part of the problem and also admitted the flaw, so they get some credit there. To end the tale on a happy note, the FBI was able to track down and catch the perpetrators, thus furthering my confidence in the Path of the Good. For some light bedtime reading, the FBI press release following the arrests of the exploiters can be found here.
Since this incident occurred, others have taken place that exploited weaknesses in support and recovery policies at other Biggies Amazon and Apple. Policies at those companies have been changed to prevent similar attacks again, though the companies seemed more concerned about inconveniencing their customers than security. While I understand that convenience is a key to keep customers coming back to your site or service, the inconvenience of having my data (or website) taken away is more important to me than the speed with which I use a service. Others seem to think so too That balance between being convenient and being secure is slowly leaning towards secure.
As a local investment advisor in my town that I like says, "Here's the Nugget":
- Be paranoid as much as is practically possible without super-pissing other people off. Inconveniencing them is OK, when done lovingly.
- Get help from and work with your vendors to help *them* make their systems better for you and others. We all benefit in the long run.
- Admit your mistakes, ask forgiveness, show how you have fixed this problem and tell why it will not be a problem again. This keeps you humble, demonstrates your commitment to your tribe and helps others, who will, in the future, and under the influence of your example, help others in return.
A friend of mine who manages a corporate office says that his users, most of them ladies, respond well to his super-paranoia because it is "sincere" paranoia and they secretly love the attention. I imagine that the conversations go something like this:
Office Lady: "Do we have to go to all of this trouble just to get our e-mail?"
Paranoid Admin: "Yes, this really is necessary to protect your data."
OL: "It seems a bit extreme."
PA: "I don't want to see you lose valuable time and information. Now hold still for your retina scan."
OL: "I really don't…"
PA: "You have very pretty eyes."
OL: "Oh. *Thank* you. Hold still like this?"