Downtime
As some of you noticed, including a number of major media organizations, WordPress.com had some unexpected downtime on Thursday evening. Whether you’re eating delicious BBQ, as I was, watching a marathon, or about to post your opus, downtime is an annoying interruption and we hate it.
This had nothing to do with our network providers, or data centers, or aliens, it was completely our fault. A single line, nay, a single character out-of-place, slipped by our normal review and testing and started overwriting settings when triggered. The team immediately took the site down to prevent further damage and clean up the mess that had been caused. All hands were called to deck.
First we determined that 11.2 million blogs were unaffected by the bug. So we brought those back up. For the remaining 50,000 or so, including some VIPs, we started restoring the lost settings using backups, audit trails, and logs. This was largely automated and we brought blogs back online as they were fixed, but a few final tricky ones were brought back one-by-one by hand because we wanted to make sure everything was in its right place.
For most folks (99%) your site was only unavailable for an hour, the rest came up a bit after that, and the tricky ones we worked on until Friday morning. Fortunately because of the time of day and the shorter duration, this had a smaller effect on traffic (about 3.9m) versus the last time (5.5m).
As a silver lining to this failing of the cloud, we learned a lot. We’ll be using our newfound experience to keep WP.com a safe, stable, and robust place to hang your hat and have your blog call home.
If you have any questions, notice any remaining wonkiness, or just want to say howdy, we’d be happy to hear from you.
- June 14, 2010
WordPress.com News 
Sometimes these things happen. Considering wordpress.com is free of charge, I personally cannot complain. Furthermore, it is damn near impossible to have 100% uptime without major expenditure by all parties concerned.
Keep up the great work, and thank you for your honesty.
John Boxall.
LikeLike
Great work getting everything back up again, already I didn’t realise anything happened at all, apart from reduction in traffic.
LikeLike
Perfect support as always and I could not even notice the downtime as I had a BBQ as well and was busy handling the hot devices and pork and beaf and drinks …
LikeLike
Thanks for fixing it in a relatively short time! I was already freaking out by not being able to access my blog! ;p
LikeLike
Thanks for your hard work! You guys are always good about helping me when my blog experiences issues.
LikeLike
I noticed it right away since I was checking stats and when I clicked on a link it was broken. I’m surprised it took this long for you to explain what happened because there were quite a few rumors going around. You’re doing the right thing by admitting your mistakes. Thanks.
LikeLike
Thanks for letting us know!
LikeLike
I didn’t notice a thing, but I’m always confident that WP sorts these things out in good time. I have no understanding of how all this techie stuff works (it’s all just words and pictures at my end), so I’m glad you people do…..
LikeLike
I appreciate your honesty, and would like to add that your actions were so swift that I personally experienced no problems.
LikeLike
Y’all have been giving us cool stuff all the time.. None of us mind these little things… Keep up the good work.. we believe in WordPress. ❤
LikeLike
all wordpress.com blogs are filtered(blocked) in Iran!
LikeLike
That’s a bummer, they should unblock us.
LikeLike
I appreciate your hard work.
LikeLike
Nice of you guys to have put that across here. I had started worrying if something had happened to my Blog Link alone. But then, the site did open up in basic format a couple of times after an hour and that assured me of all was well.
I am curious about a thing though- I’ve noticed that whenever the Site won’t open for sometime- theres some or the other feature update info released soon thereafter.
Nevertheless Thanks for having protected the users like me..
Cheers..!!
LikeLike
You guys did an amazing job. I was just in the middle of posting my blog but was amazed to receive your live twitter updates within minutes, informing me of the downtime so I could keep my “Edit Post” window open, and then as it was restored, updates on which problems your were addressing and in what order! And your transparency in these blogs really makes me appreciate how much you guys do for so many blogs! Legends. Thanks!
LikeLike
Thx for the quick repair! Sh*t just happens, you can’t do much if anything about it. I have administered high availability UNIX[tm] and also Linux servers in the past. Where the customer would lose thousands of Euro/US $ a second if the systems were down, but you have to keep calm and work on the problems…
LikeLike
Noticed it. Thanks for handling it so well.
Jamie
LikeLike
I only have one thing to say, Matt. That had to be one of the most uneventful downtime I had the pleasure of experiencing in a long long time. I followed your Twitter updates and in no time my blog was back up and running. Even though I happened to be rushing pictures from the British Forces Open Day post http://bit.ly/dr5hiC the downtime meant taking a break and making a second cup of coffee. Well done, WordPress Team. I hope to be sipping coffee with you someday.
LikeLike
I wish that many more corporates were as honest! But then stupid laws based on penalties, make admitting mistakes into a costly exercise.
LikeLike
Thank’s God, it is just temporary.
I hope next be better..
LikeLike
Good work guys. I know the feeling when something goes wrong and you’re there all night fixing it. I’m comfortable in your hands and am glad you’ve got our backs.
Jon
LikeLike
Didn’t even notice. 😛
LikeLike
Wow, what a single character out of place can do. It seems that people need to be even more careful with coding and bugs.
LikeLike
I forgive you.
LikeLike
I hope the maintenance could solve the problems. We hope the best from you guys. Thanks for the hardwork.
LikeLike
I didn’t notice anything.
LikeLike
I never realised that there had been an outage! Thanks for the info anyway!!!
LikeLike
These Things Happen WordPress still rocks
LikeLike
We hate it but can’t avoid it. Have good work WP!
LikeLike
Thank you all for your quick response and restoration. I’m in love with WordPress. If it could make my morning coffee I’d be shopping for a ring. Wonder what their answer would be. I hope it would be black, no sugar;-)
LikeLike
Hi! I really appreciate what you guys have done to make wordpress such a good platform for blogging.. Thanks! 🙂
LikeLike
Thanks !
LikeLike
Ooooh, high drama AND high tech! Geeks on red alert. Battle-stations! mass layoffs! …. you know, I think there’s some good material there for an enterprising videographer.
You run so smoothly 99.999% of the time, who can fault you for an occasional goof?
LikeLike
No layoffs. 🙂
LikeLike
No problem. Your service is impeccable otherwise, and greatly appreciated.
LikeLike
Okay, thanks for owning up and letting us know.
We forgive and still love WP
LikeLike
We love you guys!!!!! Thanks for all the hard work.
LikeLike
These things happen, can’t be helped. A swift and superb response as always : )
LikeLike
Thanks for the dedication! You do great jobs.
LikeLike
I did log in the morning and I did not have any issues with the site.
the site is seems to be working great for me 🙂
LikeLike
You guys are doing a great job! When can we expect a Facebook LIKE button?
LikeLike
Not anytime soon.
LikeLike
Thanks for being so honest and sharing what happened. Glad to be a part of the WordPress family.
LikeLike
Thanks for working so quickly to resolve the problem, and for your refreshing honesty! I really respect you guys, because you admit your mistakes when you make them.
Although aliens would have been pretty cool… 😉
LikeLike
Thanks for bringing this back up again. Sh*t happens, better luck next time.
LikeLike
I noticed the downtime, but overall was not a problem. Thanks for solving it quickly and keep up the great work.
LikeLike
Thanks for the update. I didn’t even notice. Thanks for being on top of things though and getting the problem sorted out quickly. Cheers-
LikeLike
I love the word wonkiness! Love WordPress.
LikeLike
About the downtime, will I get the notification by email when the downtime start and end ?
It could help to explain to our blog visitor I guess.
Thanks.
LikeLike
Probably not, because 1) we’re usually working 100% on bringing the site back up and 2) you couldn’t post about it anyway if it was down.
LikeLike
Thanks for keeping us posted. I was probably sleeping with WordPress went down, since I’m in Philippines & back to sleeping at night-time over here. Have you guys written AI scripts to prevent it from happening again?
LikeLike
Unfortunately we’re going to have to rely on HI (human intelligence) to keep this from happening again. But even intelligent humans make mistakes.
LikeLike
I think this problem is because aliens attack.. ha.. ha.. ha 🙂
LikeLike
Ah…I read about the downtime somewhere else too. But no problem. To err is human. 🙂
LikeLike
sad. i hate downtimes.
LikeLike
Good job guys.
LikeLike
I appreciate that you step up and say “It was our fault.” That gives a person confidence that you’ll actually fix it. (It’s useful to remind myself that, um, sometimes “It’s my fault.” I hate that.) I hated the downtime because I love, love, love my blog. So thanks for the free space for it, and thanks for getting it back up. Only an hour? Really? I was sure it was hours and hours . . . 🙂
LikeLike
I have noticed that trouble, however you guyz are still working great. We are proud of you guyz doing a lot of work at the backend.
LikeLike
I didn’t notice the actual downtime but I did notice the stats were off – only 1 visitor on Friday (hello?). I appreciate your hard work to get it back to normal. As someone else said above, you could perhaps have posted the explanation a bit sooner. You are pretty good, though, at posting notices in the help menu if a part of the system, like the stats, isn’t working.
LikeLike
I didn’t even notice since our internet provider also had a downtime! 😀
LikeLike
You are a good team and the WP blogger community has lot of faith in the WP team. Keep up the good work.
LikeLike
It’s interesting how sometimes dealing with big problems can be so stressful, but in the end make for really valuable information down the road . Not so much fun when your fighting the dragon. Glad it all worked out, back to BBQ mode. Isn’t Summer Grand!
LikeLike
This downtime wasn’t even a blip on my radar, I was asleep. But reading Techcrunch’s post about it over my morning coffee had me choking with laughter. I nearly reblogged it. 🙂
LikeLike
If only the politicians were this honest :).
LikeLike
Good job! You guys are the reason why I love wordpress so much =)
LikeLike
I so appreciate how you handled this incident. I also strongly feel that you are superb professionals and took the necessary measures to rectify the problem and let us know in a timely manner. I commend you!
LikeLike
I didn’t realise that WP was down, but it is very good to know that there is a support team out there 🙂
LikeLike
In the big picture this was definitely not a biggie 🙂 For those that don’t say it, thank you for all you do, all you offer and include for basically a free service! As a previous small business owner paying for hosting and templates, etc. WordPress is a breath of fresh air and a nice relief to my pocketbook to boot. And as a reminder to others out there, even people you pay for hosting has downtime, unexpected and expected and not as great of customer service as you guys do. Good job an keep it up!
LikeLike
Yes it was quite puzzling as wordpress is soooo reliable. I googled to find an answer to the mystery that night but nothing there. Thanks for the explanation. What are we to do without our wordpress?
LikeLike
As for me, I think that the WordPress team does exceptional work. Every blogging platform experiences unexpected downtimes that cannot be avoided. You all ought to be commended and thanked for the most excellent job that you do for this blogging community. Really really.
-Lorna
LikeLike
i am really happy you take this so seriously. some one rightly said , the best things in life are free
LikeLike
Thanks for the transparency, Matt. I thought Tech Crunch was a little harsh on you all over the downtime while you’re in the center of it struggling to get out. I’m sure that public pricking stung a bit.
This is precisely why I moved 13 blogs from my own Movable Type standalone installation back to WordPress.com — every blog inevitably goes down — and what matters is how the blog gets back up and how you make that healing happen.
On WordPress.com — something breaks — I know it will be fixed and all I have to do is wait.
When you’re alone, and on your own, if it breaks, you fix it — even if you didn’t break it! One time, after a Movable Type upgrade, every time I would click the “READ MORE” link on my blogs, the click would come up on a blank page. I was going crazy for a day trying to figure what broke and who broke it and why.
Finally, someone told me I’d munged one of my core templates — and I had — I pasted content from one template wrongly into another and the cascading effect was absolutely ridiculous and numbing and 100% my fault. I broke my blogs big time!
So… I say… “Better you than me” — when it comes to fixing broken things — because I know you and your team together know much more about fixing this sort of stuff than I ever could alone.
Thank you for WordPress.com.
LikeLike
It wasn’t a HUGE biggy, except I did get a little mad when I couldn’t axcess my blog, either way, thanks for the heads up on what happened, by the way, is there any chance that there will be any tools to fix your header while uploading it? Maybe like borders or something?
LikeLike
There’s that old time saying, “Live and learn”….. I don’t know who first said it, but WordPress takes it seriously. 🙂 … Thank you for your dedication in staying on top of things.
LikeLike
Mistakes happen some times. No one can be 100% perfect. Thanks for all hands on deck getting us up and going again. Thanks to WordPress for your terrific work.
LikeLike
Thanks! Wish I could handle all of my downtime with the same skill and finesse. LOL
LikeLike
Thanks for your service, and I can’t believe how fast you got all those blogs back up and running. Good work.
LikeLike
Speaking for myself, what matters is the reaction, and until I read about it here, I was blissfully ignorant of the event, so as far as I’m concerned, you handled it in a manner that made it transparent to me.
Thanks!
LikeLike
I didn’t realize the down time either…
if it had been my email, I’d probably had noticed.
Thanks for letting us know.
LikeLike
Thank you for sorting out the problem quickly!
LikeLike
WordPress–keep up the good work. We expect there to be glitches every now and then—the main thing is—wordpress fixes them. WordPress.com rocks on customer support.
LikeLike
What did you have on the grill?
LikeLike
You do a terrific job and frankly make blogspot look hopelessly lame. More power to you.
LikeLike
I didn’t notice it. All I can say is… I am so impressed with WordPress.com. Your site makes blogging easy and fun.. keep up the good work!
LikeLike
Hello there.
I was surprised thursday night when I found the site offline. I got a bit annoyed then because I just had recommended WordPress.com to a friend who has been a long-time Typepad user. Anyway, your honesty is really remarkable. It is really a pleasure to use (and recommend) your service.
LikeLike
The fact that you reported this and took responsibility for it, says a lot about you at WordPress! Superior Customer Care – that’s WordPress in the nutshell!
Thank You!
ATW
LikeLike
Hats off to you guys. Not just for the good work, the excellent service you provide or the dilligence you do it with,… but also for not hiding behind excuses. Coming right out with a hand raised to claim the mistake is an admirable trait in anyone – and to come out doing it in THIS way with no BS, telling it like it is… that inspires confidence and trust. If I ever need a reminder of why I host my blogs here… this will be a perfect example.
Thank you for all your hard work! It is SO appreciated.
LikeLike
I really appreciate that WordPress keeps their users up to date on information and makes no excuses about issues they have. You guys are candid and down-to-earth and I thank you for that. Keep up the great work. : )
LikeLike
Thanks for your clearity about distortions, someone who acknowledges honestly his own hickup’s is to be respected! Personally, I had no pain with it. Keep on going strong! Greets,
Rob
LikeLike
Well, I didn’t notice anything, but you guys have done a great job!
LikeLike
Never noticed.
LikeLike
There was downtime? I’ve got to spend less time outside 🙂
LikeLike
I didn’t know that anything had happened. Thank you for letting us know and promptly fixing it. Good work!
LikeLike
Things went very smoothly on this end and I really appreciate your public announcement.
LikeLike
Thanks for the honest, straightforward answer. I love WordPress and the fact that your willing to admit your mistakes to your clients is very reassuring. Honestly, the fact that you were willing to pull down the entire system to make sure the problems was fixed with limited data loss speaks volumes about your dedication to customer service. Thanks for running such a great site!
LikeLike
Thanks for the update! 🙂
LikeLike
Appreciated, thanks.
LikeLike
Well I certainly did not notice anything but applaud all concerned for your sheer hard work in fixing things with your usual speed and efficiency but mostly for being up front and honest and for keeping us informed from the beginning. Well done guys and thank you.
If only a certain Broadband/TV provider would take a leaf out of your book!!
LikeLike
You guys always did a wonderful Jobs. Thank You!
LikeLike
I appreciate that you honesty let us know. Thank you.
LikeLike